Runtime program regression analysis tool for a simulation engine

ABSTRACT

A system is disclosed that provides a goal based learning system utilizing a rule based expert training system to provide a cognitive educational experience. The system provides the user with a simulated environment that presents a business opportunity to understand and solve optimally. Mistakes are noted and remedial educational material presented dynamically to build the necessary skills that a user requires for success in the business endeavor. The system utilizes an artificial intelligence engine driving individualized and dynamic feedback with synchronized video and graphics used to simulate real-world environment and interactions. Multiple “correct” answers are integrated into the learning system to allow individualized learning experiences in which navigation through the system is at a pace controlled by the learner. A robust business model provides support for realistic activities and allows a user to experience real world consequences for their actions and decisions and entails realtime decision-making and synthesis of the educational material. The system includes tools for analysis, logging and display of regression analysis information for a presentation as it is presented.

FIELD OF THE INVENTION

The present invention relates to education systems and more particularlyto a rule based tutorial system which includes analysis, logging anddisplay of regression analysis information for a presentation as it ispresented during business simulations of actual environments to teachnew skills.

BACKGROUND OF THE INVENTION

When building a knowledge based system or expert system, at least twodisciplines are necessary to properly construct the rules that drive theknowledge base, the discipline of the knowledge engineer and theknowledge of the expert. The domain expert has knowledge of the domainor field of use of the expert system. For example, the domain expert ofan expert for instructing students in an automotive manufacturingfacility might be a process control engineer while the domain expert fora medical instruction system might be a doctor or a nurse. The knowledgeengineer is a person that understands the expert system and utilizes theexpert's knowledge to create an application for the system. In manyinstances, the knowledge engineer and domain expert are separate peoplewho have to collaborate to construct the expert system.

Typically, this collaboration takes the form of the knowledge engineerasking questions of the domain expert and incorporating the answers tothese questions into the design of the system. This approach is laborintensive, slow and error prone. The coordination of the two separatedisciplines may lead to problems. Although the knowledge engineer cantranscribe input from the expert utilizing videotape, audio tape, textand other sources, efforts from people of both disciplines have to beexpended. Further, if the knowledge engineer does not ask the rightquestions or asks the questions in an incorrect way, the informationutilized to design the knowledge base could be incorrect. Feedback tothe knowledge engineer from the expert system is often not available inprior art system until the construction is completed. With conventionalsystem, there is a time consuming feedback loop that ties togethervarious processes from knowledge acquisition to validation.

Educational systems utilizing an expert system component often sufferfrom a lack of motivational aspects that result in a user becoming boredor ceasing to complete a training program. Current training programsutilize static, hard-coded feedback with some linear video and graphicsused to add visual appeal and illustrate concepts. These systemstypically support one “correct” answer and navigation through the systemis only supported through a single defined path which results in atwo-dimensional generic interaction, with no business model support anda single feedback to the learner of correct or incorrect based on theselected response. Current tutorial systems do not architect realbusiness simulations into the rules to provide a creative learningenvironment to a user.

SUMMARY OF THE INVENTION

According to a broad aspect of a preferred embodiment of the invention,a goal based learning system utilizes a rule based expert trainingsystem to provide a cognitive educational experience. The systemprovides the user with a simulated environment that presents a businessopportunity to understand and solve optimally. Mistakes are noted andremedial educational material presented dynamically to build thenecessary skills that a user requires for success in the businessendeavor. The system utilizes an artificial intelligence engine drivingindividualized and dynamic feedback with synchronized video and graphicsused to simulate real-world environment and interactions. Multiple“correct” answers are integrated into the learning system to allowindividualized learning experiences in which navigation through thesystem is at a pace controlled by the learner. A robust business modelprovides support for realistic activities and allows a user toexperience real world consequences for their actions and decisions andentails realtime decision-making and synthesis of the educationalmaterial. The system includes tools for analysis, logging and display ofregression analysis information for a presentation as it is presented.

DESCRIPTION OF THE DRAWINGS

The foregoing and other objects, aspects and advantages are betterunderstood from the following detailed description of a preferredembodiment of the invention with reference to the drawings, in which:

FIG. 1 is a block diagram of a representative hardware environment inaccordance with a preferred embodiment;

FIG. 2 is a block diagram of a system architecture in accordance with apreferred embodiment;

FIG. 3 depicts the timeline and relative resource requirements for eachphase of development for a typical application development in accordancewith a preferred embodiment;

FIG. 4 illustrates a small segment of a domain model for claims handlersin the auto insurance industry in accordance with a preferredembodiment;

FIG. 5 illustrates an insurance underwriting profile in accordance witha preferred embodiment;

FIG. 6 illustrates a transformation component in accordance with apreferred embodiment;

FIG. 7 illustrates the use of a toolbar to navigate and accessapplication level features in accordance with a preferred embodiment;

FIG. 8 is a GBS display in accordance with a preferred embodiment;

FIG. 9 is a feedback display in accordance with a preferred embodiment;

FIG. 10 illustrates a journal entry simulation in accordance with apreferred embodiment;

FIG. 11 illustrates a simulated Bell Phone Bill journal entry inaccordance with a preferred embodiment;

FIG. 12 illustrates a feedback display in accordance with a preferredembodiment;

FIG. 13 illustrates the steps of the first scenario in accordance with apreferred embodiment;

FIGS. 14 and 15 illustrate the steps associated with a build scenario inaccordance with a preferred embodiment;

FIG. 16 illustrates a test scenario in accordance with a preferredembodiment. The test students work through the journalization activity;

FIG. 17 illustrates how the tool suite supports student administrationin accordance with a preferred embodiment;

FIG. 18 illustrates a suite to support a student interaction inaccordance with a preferred embodiment;

FIG. 19 illustrates the remediation process in accordance with apreferred embodiment;

FIG. 20 illustrates the objects for the journalization task inaccordance with a preferred embodiment;

FIG. 21 illustrates the mapping of a source item to a target item inaccordance with a preferred embodiment;

FIG. 22 illustrates an analysis of rules in accordance with a preferredembodiment;

FIG. 23 illustrates a feedback selection in accordance with a preferredembodiment;

FIG. 24 is a flowchart of the feedback logic in accordance with apreferred embodiment;

FIG. 25 is a block diagram setting forth the architecture of asimulation model in accordance with a preferred embodiment;

FIG. 26 illustrates the steps for configuring a simulation in accordancewith a preferred embodiment;

FIG. 27 is a block diagram presenting the detailed architecture of asystem dynamics model in accordance with a preferred embodiment;

FIG. 28 is an overview diagram of the logic utilized for initialconfiguration in accordance with a preferred embodiment;

FIG. 29 is a display of video information in accordance with a preferredembodiment; and

FIG. 30 illustrates an ICA utility in accordance with a preferredembodiment.

DETAILED DESCRIPTION

A preferred embodiment of a system in accordance with the presentinvention is preferably practiced in the context of a personal computersuch as an IBM compatible personal computer, Apple Macintosh computer orUNIX based workstation. A representative hardware environment isdepicted in FIG. 1, which illustrates a typical hardware configurationof a workstation in accordance with a preferred embodiment having acentral processing unit 110, such as a microprocessor, and a number ofother units interconnected via a system bus 112. The workstation shownin FIG. 1 includes a Random Access Memory (RAM) 114, Read Only Memory(ROM) 116, an I/O adapter 118 for connecting peripheral devices such asdisk storage units 120 to the bus 112, a user interface adapter 122 forconnecting a keyboard 124, a mouse 126, a speaker 128, a microphone 132,and/or other user interface devices such as a touch screen (not shown)to the bus 112, communication adapter 134 for connecting the workstationto a communication network (e.g., a data processing network) and adisplay adapter 136 for connecting the bus 112 to a display device 138.The workstation typically has resident thereon an operating system suchas the Microsoft Windows NT or Windows/95 Operating System (OS), the IBMOS/2 operating system, the MAC OS, or UNIX operating system. Thoseskilled in the art will appreciate that the present invention may alsobe implemented on platforms and operating systems other than thosementioned.

A preferred embodiment is written using JAVA, C, and the C++ languageand utilizes object oriented programming methodology. Object orientedprogramming (OOP) has become increasingly used to develop complexapplications. As OOP moves toward the mainstream of software design anddevelopment, various software solutions require adaptation to make useof the benefits of OOP. A need exists for these principles of OOP to beapplied to a messaging interface of an electronic messaging system suchthat a set of OOP classes and objects for the messaging interface can beprovided. A simulation engine in accordance with a preferred embodimentis based on a Microsoft Visual Basic component developed to help designand test feedback in relation to a Microsoft Excel spreadsheet. Thesespreadsheet models are what simulate actual business functions andbecome a task that will be performed by a student The Simulation Engineaccepts simulation inputs and calculates various outputs and notifiesthe system of the status of the simulation at a given time in order toobtain appropriate feedback.

Relationship of Components

The simulation model executes the business function that the student islearning and is therefore the center point of the application. Anactivity ‘layer’ allows the user to visually guide the simulation bypassing inputs into the simulation engine and receiving an output fromthe simulation model. For example, if the student was working on anincome statement activity, the net sales and cost of goods soldcalculations are passed as inputs to the simulation model and the netincome value is calculated and retrieved as an output. As calculationsare passed to and retrieved from the simulation model, they are alsopassed to the Intelligent Coaching Agent (ICA). The ICA analyzes theInputs and Outputs to the simulation model and generates feedback basedon a set of rules. This feedback is received and displayed through theVisual Basic Architecture.

FIG. 2 is a block diagram of a system architecture in accordance with apreferred embodiment. The Presentation “layer” 210 is separate from theactivity “layer” 220 and communication is facilitated through a SystemDynamics Engine 230 that control the display specific content topics. Apreferred embodiment enables knowledge workers 200 & 201 to acquirecomplex skills rapidly, reliably and consistently across an organizationto deliver rapid acquisition of complex skills. This result is achievedby placing individuals in a simulated business environment that “looksand feels” like real work, and challenging them to make decisions whichsupport a business” strategic objectives utilizing highly effectivelearning theory (e.g., goal based learning, learn by doing, failurebased learning, etc.), and the latest in multimedia user interfaces,coupled with three powerful, integrated software components. The firstof these components is the Systems Dynamics Engine 230 consisting of amathematical modeling tool which simulates business outcomes of anindividual's collective actions over a period of time. The secondcomponent is a System Dynamics Model 250 consisting of an HTML contentlayer which organizes and presents packaged knowledge much like anonline text book with practice exercises, video war stories, and aglossary. The third component is an Intelligent Coaching Agent 270comprising a Simulation Engine 240 which generates individualizedcoaching messages based on decisions made by learner.

Feedback is unique for each individual completing the course andsupports Deliver Feedback 242 “designed into” the course. A businesssimulation methodology that includes support for content acquisition,story line design, interaction design, feedback and coaching delivery,and content delivery is architected into the system in accordance with apreferred embodiment. A large number of “pre-designed” learninginteractions such as drag and drop association of Inputs Outputs 238,situation assessment/action planning, interviewing (one-on-one,one-to-many), presenting (to a group of experts/executives), metering ofperformance (handle now, handle later), “time jumping” for impact ofdecisions, competitive landscape shift (while “time jumping”,competitors merge, customers are acquired, etc.) and video interviewingwith automated note taking are also included in accordance with apreferred embodiment.

Business simulation in accordance with a preferred embodiment deliverstraining curricula in an optimal manner. This is because suchapplications provide effective training that mirrors a student's actualwork environment. The application of skills “on the job” facilitatesincreased retention and higher overall job performance. While theresults of such training applications are impressive, businesssimulations are very complex to design and build correctly. Thesesimulations are characterized by a very open-ended environment, wherestudents can go through the application along any number of paths,depending on their learning style and prior experiences/knowledge.

A category of learning approaches called Learn by Doing, is commonlyused as a solution to support the first phase (Learn) of the WorkforcePerformance Cycle. However, it can also be a solution to support thesecond phase (Perform) of the cycle to enable point of need learningduring job performance. By adopting the approach presented, some of thebenefits of a technology based approach for building business simulationsolutions which create more repeatable, predictable projects resultingin more perceived and actual user value at a lower cost and in less timeare highlighted.

Most corporate training programs today are misdirected because they havefailed to focus properly on the purpose of their training. Theseprograms have confused the memorization of facts with the ability toperform tasks; the knowing of “that” with the knowing of “how”. Byadopting the methods of traditional schools, businesses are teaching awide breadth of disconnected, decontextualized facts and figures, whenthey should be focused on improved performance. How do you teachperformance, when lectures, books, and tests inherently are designedaround facts and figures? Throw away the lectures, books, and tests. Thebest way to prepare for high performance is to perform; experience isthe best teacher! Most business leaders agree that workers become moreeffective the more time they spend in their jobs. The best approach fortraining novice employees, therefore, would be letting them learn on thejob, acquiring skills in their actual work environment. The idea oflearning-by-doing is not revolutionary, yet it is resisted in businessand academia. Why is this so, if higher competence is universallydesired?

Learners are reluctant to adopt learning-by-doing because they arefrightened of failure. People work hard to avoid making mistakes infront of others. Business leaders are hesitant to implementlearning-by-doing because novice failure may have dramatic safety, legaland financial implications. Imagine a novice pilot learning-by-doing ashe accelerates a large jet plane down a runway; likewise, consider a newfinancial analyst learning-by-doing as he structures a multi-milliondollar financial loan. Few employers are willing to endure such failuresto have a more competent workforce.

The key to such a support system is that it is seamlessly integratedinto the business system that the knowledge worker uses to execute theirjob tasks. Workers don't need to go “off-line” or seek out crypticinformation buried within paper manuals and binders for guidance or tofind the answer to queries. All the support components are madeavailable through the same applications the worker's use, at the pointin which they need them, tailored to the individual to show “how”, notjust “what”. Learning would be occurring all the time, with littledistinction between performing and improving performance. Establishingthat training should focus on performance (how), rather than facts(what), and extending the model of learning to include assistance whileperforming, rather than only before performance, still leaves usdangerously exposed in preparing to compete in the new, chaotic economy.As was mentioned in the opening of this paper, the pace of change inbusiness today is whiplash fast. Not only are new methods of doingbusiness evolving every 18-24 months, new competitors emerge, dominate,and fade in time periods businesses used to take to perform demographicstudies. Now more than ever, those who do not reinvent themselves on aregular basis will be fossilized by the pace of change. A typical BusSimengagement takes between one and two years to complete and requires avariety of both functional and technical skills. FIG. 3 depicts thetimeline and relative resource requirements for each phase ofdevelopment for a typical application development in accordance with apreferred embodiment. The chart clearly depicts the relationship betweenthe large number of technical resources required for both the build andtest phases of development. This is because the traditional developmentprocess used to build BusSim solutions reflects more of a “one off”philosophy, where development is done from scratch in a monolithicfashion, with little or no reuse from one application to the next. Thislack of reuse makes this approach prohibitively expensive, as well aslengthy, for future BusSim projects.

The solution to this problem is to put tools in the hands ofinstructional designers that allows them to create their BusSim designsand implement them without the need for programmers to write code. Andto put application architectures that integrate with the tools in thehands of developers, providing them with the ability to quickly deliversolutions for a number of different platforms. The reuse, then, comes inusing the tools and architectures from one engagement to another. Bothfunctional and technical resources carry with them the knowledge of howto use the technology, which also has an associated benefit ofestablishing a best-practice development methodology for BusSimengagements.

Development Cycle Activities

In the Design Phase, instructional designers become oriented to thecontent area and begin to conceptualize an instructional approach. Theyfamiliarize themselves with the subject matter through reading materialsand interviews with Subject Matter Experts (SMEs). They also identifylearning objectives from key client contacts. Conceptual designs forstudent interactions and interface layouts also begin to emerge. Afterthe conceptual designs have taken shape, Low-Fi user testing (a.k.a.Conference Room Piloting) is performed. Students interact with interfacemock-ups while facilitators observe and record any issues. Finally,detailed designs are created that incorporate findings. These detaileddesigns are handed off to the development team for implementation. Thedesign phase has traditionally been fraught with several problems.Unlike a traditional business system, BusSim solutions are not rooted intangible business processes, so requirements are difficult to identifyin a concrete way. This leaves instructional designers with a ‘blue sky’design problem. With few business-driven constraints on the solution,shallow expertise in the content area, and limited technical skills,instructional designers have little help in beginning a design.Typically, only experienced designers have been able to conjureinterface, analysis, and feedback designs that meet the learningobjectives yet remain technically feasible to implement. To compound theproblem, BusSim solutions are very open ended in nature. The designermust anticipate a huge combination of student behavior to designfeedback that is helpful and realistic.

During the build phase, the application development team uses thedetailed designs to code the application. Coding tasks include theinterfaces and widgets that the student interacts with. The interfacescan be made up of buttons, grids, check boxes, or any other screencontrols that allow the student to view and manipulate his deliverables.The developer must also code logic that analyzes the student's work andprovides feedback interactions. These interactions may take the form oftext and/or multimedia feedback from simulated team members,conversations with simulated team members, or direct manipulations ofthe student's work by simulated team members. In parallel with thesecoding efforts, graphics, videos, and audio are being created for use inthe application. Managing the development of these assets have their owncomplications. Risks in the build phase include misinterpretation of thedesigns. If the developer does not accurately understand the designer'sintentions, the application will not function as desired. Also, codingthese applications requires very skilled developers because the logicthat analyzes the student's work and composes feedback is very complex.

The Test Phase, as the name implies, is for testing the application.Testing is performed to verify the application in three ways: first thatthe application functions properly (functional testing), second that thestudents understand the interface and can navigate effectively(usability testing), and third that the learning objectives are met(cognition testing). Functional testing of the application can becarried out by the development team or by a dedicated test team. If theapplication fails to function properly, it is debugged, fixed,recompiled and retested until its operation is satisfactory. Usabilityand cognition testing can only be carried out by test students who areunfamiliar with the application. If usability is unsatisfactory, partsof the interface and or feedback logic may need to be redesigned,recoded, and retested. If the learning objectives are not met, largeparts of the application may need to be removed and completelyredeveloped from a different perspective. The test phase is typicallywhere most of the difficulties in the BusSim development cycle areencountered. The process of discovering and fixing functional,usability, and cognition problems is a difficult process and not anexact science.

For functional testing, testers operate the application, either byfollowing a test script or by acting spontaneously and documenting theiractions as they go. When a problem or unexpected result is encountered,it too is documented. The application developer responsible for thatpart of the application then receives the documentation and attempts toduplicate the problem by repeating the tester's actions. When theproblem is duplicated, the developer investigates further to find thecause and implement a fix. The developer once again repeats the tester'sactions to verify that the fix solved the problem. Finally, all othertest scripts must be rerun to verify that the fix did not haveunintended consequences elsewhere in the application. The ExecutionPhase refers to the steady state operation of the completed applicationin its production environment. For some clients, this involves phonesupport for students. Clients may also want the ability to trackstudents' progress and control their progression through the course.Lastly, clients may want the ability to track issues so they may beconsidered for inclusion in course maintenance releases.

One of the key values of on-line courses is that they can be taken at atime, location, and pace that is convenient for the individual student.However, because students are not centrally located, support is notalways readily available. For this reason it is often desirable to havephone support for students. Clients may also desire to track students'progress, or control their advancement through the course. Under thisstrategy, after a student completes a section of the course, he willtransfer his progress data to a processing center either electronicallyor by physically mailing a disk. There it can be analyzed to verify thathe completed all required work satisfactorily. One difficulty commonlyassociated with student tracking is isolating the student data foranalysis. It can be unwieldy to transmit all the course data, so it isoften imperative to isolate the minimum data required to perform thenecessary analysis of the student's progress.

A Delivery Framework for Business Simulation

As discussed earlier, the traditional development process used to buildBusSim solutions reflects more of a “one off” philosophy, wheredevelopment is done from scratch in a monolithic fashion, with little orno reuse from one application to the next. A better approach would be tofocus on reducing the total effort required for development throughreuse, which, in turn would decrease cost and development time. Thefirst step in considering reuse as an option is the identification ofcommon aspects of the different BusSim applications that can begeneralized to be useful in future applications. In examination of theelements that make up these applications, three common aspects emerge asintegral parts of each: Interface, Analysis and Interpretation. EveryBusSim application must have a mechanism for interaction with thestudent. The degree of complexity of each interface may vary, from thehigh interactivity of a high-fidelity real-time simulation task, to theless complex information delivery requirements of a business casebackground information task. Regardless of how sophisticated the UserInterface (UI), it is a vital piece of making the underlying simulationand feedback logic useful to the end user.

Every BusSim application does analysis on the data that defines thecurrent state of the simulation many times throughout the execution ofthe application. This analysis is done either to determine what ishappening in the simulation, or to perform additional calculations onthe data which are then fed back into the simulation. For example, theanalysis may be the recognition of any actions the student has taken onartifacts within the simulated environment (notebooks, number values,interviews conducted, etc.), or it may be the calculation of an ROIbased on numbers the student has supplied. Substantive, useful feedbackis a critical piece of any BusSim application. It is the main mechanismto communicate if actions taken by the student are helping or hurtingthem meet their performance objectives. The interpretation piece of theset of proposed commonalties takes the results of any analysis performedand makes sense of it. It takes the non-biased view of the world thatthe Analysis portion delivers (i.e., “Demand is up 3%”) and places someevaluative context around it (i.e., “Demand is below the expected 7%;you're in trouble!”, or “Demand has exceeded projections of 1.5%; Greatjob!”).

There are several approaches to capturing commonalties for reuse. Two ofthe more common approaches are framework-based and component-based. Tohelp illustrate the differences between the two approaches, we will drawan analogy between building an application and building a house. One canconstruct a house from scratch, using the raw materials, 2×4s, nails,paint, concrete, etc. One can also construct an application fromscratch, using the raw materials of new designs and new code. The effortinvolved in both undertakings can be reduced through framework-basedand/or component-based reuse. Within the paradigm of framework-basedreuse, a generic framework or architecture is constructed that containscommonalties. In the house analogy, one could purchase a prefabricatedhouse framework consisting of floors, outside walls, bearing walls and aroof. The house can be customized by adding partition walls, wall-paper,woodwork, carpeting etc. Similarly, prefabricated application frameworksare available that contain baseline application structure andfunctionality. Individual applications are completed by adding specificfunctionality and customizing the look-and-feel. An example of acommonly used application framework is Microsoft Foundation Classes. Itis a framework for developing Windows applications using C++. MFCsupplies the base functionality of a windowing application and thedeveloper completes the application by adding functionality within theframework. Framework-based reuse is best suited for capturingtemplate-like features, for example user interface management,procedural object behaviors, and any other features that may requirespecialization. Some benefits of using a framework include:

Extensive functionality can be incorporated into a framework. In thehouse analogy, if I know I am going to build a whole neighborhood ofthree bedroom ranches, I can build the plumbing, wiring, and partitionwalls right into the framework, reducing the incremental effort requiredfor each house. If I know I am going to build a large number of verysimilar applications, they will have more commonalties that can beincluded in the framework rather than built individually.

Applications can override the framework-supplied functionality whereverappropriate. If a house framework came with pre-painted walls, thebuilder could just paint over them with preferred colors. Similarly, theobject oriented principle of inheritance allows an application developerto override the behavior of the framework. In the paradigm ofcomponent-based reuse, key functionality is encapsulated in a component.The component can then be reused in multiple applications. In the houseanalogy, components correspond to appliances such as dishwashers,refrigerators, microwaves, etc. Similarly, many application componentswith pre-packaged functionality are available from a variety of vendors.An example of a popular component is a Data Grid. It is a component thatcan be integrated into an application to deliver the capability ofviewing columnar data in a spreadsheet-like grid. Component-based reuseis best suited for capturing black-box-like features, for example textprocessing, data manipulation, or any other features that do not requirespecialization.

Several applications on the same computer can share a single component.This is not such a good fit with the analogy, but imagine if all thehouses in a neighborhood could share the same dishwasher simultaneously.Each home would have to supply its own dishes, detergent, and water, butthey could all wash dishes in parallel. In the application componentworld, this type of sharing is easily accomplished and results inreduced disk and memory requirements.

Components tend to be less platform and tool dependent. A microwave canbe used in virtually any house, whether it's framework is steel or wood,and regardless of whether it was customized for building mansions orshacks. You can put a high-end microwave in a low-end house andvice-versa. You can even have multiple different microwaves in yourhouse. Component technologies such as CORBA, COM, and Java Beans makethis kind of flexibility commonplace in application development. Often,the best answer to achieving reuse is through a combination offramework-based and component-based techniques. A framework-basedapproach for building BusSim applications is appropriate for developingthe user interface, handling user and system events, starting andstopping the application, and other application-specific and deliveryplatform-specific functions. A component-based approach is appropriatefor black-box functionality. That is, functionality that can be usedas-is with no specialization required. In creating architectures tosupport BusSim application development, it is imperative that any assetsremain as flexible and extensible as possible or reusability may bediminished. Therefore, we chose to implement the unique aspects ofBusSim applications using a component approach rather than a frameworkapproach. This decision is further supported by the followingobservations.

Delivery Framework for Business Simulation

Components are combined with an Application Framework and an ApplicationArchitecture to achieve maximum reuse and minimum custom developmenteffort. The Application Architecture is added to provide communicationsupport between the application interface and the components, andbetween the components. This solution has the following features: Thecomponents (identified by the icons) encapsulate key BusSimfunctionality. The Application Architecture provides the glue thatallows application-to-component and component-to-componentcommunication. The Application Framework provides structure and basefunctionality that can be customized for different interaction styles.Only the application interface must be custom developed. The nextsection discusses each of these components in further detail.

The Business Simulation Toolset

We have clearly defined why a combined component/framework approach isthe best solution for delivering high-quality BusSim solutions at alower cost. Given that there are a number of third party frameworksalready on the market that provide delivery capability for a widevariety of platforms, the TEL project is focused on defining anddeveloping a set of components that provide unique services for thedevelopment and delivery of BusSim solutions. These components alongwith a set of design and test workbenches are the tools used byinstructional designers to support activities in the four phases ofBusSim development. We call this suite of tools the Business SimulationToolset. Following is a description of each of the components andworkbenches of the toolset. A Component can be thought of as a black boxthat encapsulates the behavior and data necessary to support a relatedset of services. It exposes these services to the outside world throughpublished interfaces. The published interface of a component allows youto understand what it does through the services it offers, but not howit does it. The complexity of its implementation is hidden from theuser. The following are the key components of the BusSim Toolset. DomainComponent—provides services for modeling the state of a simulation.Profiling Component—provides services for rule-based evaluating thestate of a simulation. Transformation Component—provides services formanipulating the state of a simulation. Remediation Component—providesservices for the rule-based delivering of feedback to the student TheDomain Model component is the central component of the suite thatfacilitates communication of context data across the application and theother components. It is a modeling tool that can use industry-standarddatabase such as Informix, Oracle, or Sybase to store its data. A domainmodel is a representation of the objects in a simulation. The objectsare such pseudo tangible things as a lever the student can pull, a formor notepad the student fills out, a character the student interacts within a simulated meeting, etc. They can also be abstract objects such asthe ROI for a particular investment, the number of times the studentasked a particular question, etc. These objects are called entities.Some example entities include: Vehicles, operators and incidents in aninsurance domain; Journal entries, cash flow statements and balancesheets in a financial accounting domain and Consumers and purchases in amarketing domain.

An entity can also contain other entities. For example, a personal bankaccount entity might contain an entity that represents a savingsaccount. Every entity has a set of properties where each property insome way describes the entity. The set of properties owned by an entity,in essence, define the entity. Some example properties include: Anincident entity on an insurance application owns properties such as“Occurrence Date”, “Incident Type Code”, etc. A journal entry ownsproperties such as “Credit Account”, “Debit Account”, and “Amount”; anda revolving credit account entity on a mortgage application ownsproperties such as “Outstanding Balance”, “Available Limit”, etc. FIG. 4illustrates a small segment of a domain model for claims handlers in theauto insurance industry in accordance with a preferred embodiment.

Profiling Component

In the simplest terms, the purpose of the Profiling Component is toanalyze the current state of a domain and identify specific things thatare true about that domain. This information is then passed to theRemediation Component which provides feedback to the student. TheProfiling Component analyzes the domain by asking questions about thedomain's state, akin to an investigator asking questions about a case.The questions that the Profiler asks are called profiles. For example,suppose there is a task about building a campfire and the student hasjust thrown a match on a pile of wood, but the fire didn't start. Inorder to give useful feedback to the student, a tutor would need to knowthings like: was the match lit?, was the wood wet?, was there kindlingin the pile?, etc. These questions would be among the profiles that theProfiling Component would use to analyze the domain. The results of theanalysis would then be passed off to the Remediation Component whichwould use this information to provide specific feedback to the student.Specifically, a profile is a set of criteria that is matched against thedomain. The purpose of a profile is to check whether the criteriadefined by the profile is met in the domain. Using a visual editingtool, instructional designers create profiles to identify those thingsthat are important to know about the domain for a given task. Duringexecution of a BusSim application at the point that feedback isrequested either by the student or pro-actively by the application, theset of profiles associated with the current task are evaluated todetermine which ones are true. Example profiles include: Goodproductions strategy but wrong Break-Even Formula; Good driving recordand low claims history; and Correct Cash Flow Analysis but poor Returnon Investment (ROI)

A profile is composed of two types of structures: characteristics andcollective characteristics. A characteristic is a conditional (the ifhalf of a rule) that identifies a subset of the domain that is importantfor determining what feedback to deliver to the student. Examplecharacteristics include: Wrong debit account in transaction 1; Perfectcost classification; At Least 1 DUI in the last 3 years; More than $4000in claims in the last 2 years; and More than two at-fault accidents in 5years A characteristic's conditional uses one or more atomics as theoperands to identify the subset of the domain that defines thecharacteristic. An atomic only makes reference to a single property of asingle entity in the domain; thus the term atomic. Example atomicsinclude: The number of DUI's>=1; ROI>10%; and Income between $75,000 and$110,000. A collective characteristic is a conditional that usesmultiple characteristics and/or other collective characteristics as itsoperands. Collective characteristics allow instructional designers tobuild richer expressions (i.e., ask more complex questions). Examplecollective characteristics include: Bad Household driving record; GoodCredit Rating; Marginal Credit Rating; Problems with Cash for Expensetransactions; and Problems with Sources and uses of cash. Once created,designers are able to reuse these elements within multiple expressions,which significantly eases the burden of creating additional profiles.When building a profile from its elements, atomics can be used bymultiple characteristics, characteristics can be used by multiplecollective characteristics and profiles, and collective characteristicscan be used by multiple collective characteristics and profiles. FIG. 5illustrates an insurance underwriting profile in accordance with apreferred embodiment.

Example Profile for Insurance Underwriting

Transformation Component—Whereas the Profiling Component asks questionsabout the domain, the Transformation Component performs calculations onthe domain and feeds the results back into the domain for furtheranalysis by the Profiling Component. This facilitates the modeling ofcomplex business systems that would otherwise be very difficult toimplement as part of the application. Within the Analysis phase of theInterface/Analysis/Interpretation execution flow, the TransformationComponent actually acts on the domain before the Profiling Componentdoes its analysis. The Transformation Component acts as a shell thatwraps one or more data modeling components for the purpose ofintegrating these components into a BusSim application. TheTransformation Component facilitates the transfer of specific data fromthe domain to the data modeling component (inputs) for calculations tobe performed on the data, as well as the transfer of the results of thecalculations from the data modeling component back to the domain(outputs). FIG. 6 illustrates a transformation component in accordancewith a preferred embodiment. The data modeling components could be thirdparty modeling environments such as spreadsheet-based modeling (e.g.,Excel, Formula1) or discrete time-based simulation modeling (e.g.,PowerSim, VenSim). The components could also be custom built in C++, VB,Access, or any tool that is ODBC compliant to provide unique modelingenvironments. Using the Transformation Component to wrap a third partyspreadsheet component provides an easy way of integrating into anapplication spreadsheet-based data analysis, created by such tools asExcel. The Transformation Component provides a shell for the spreadsheetso that it can look into the domain, pull out values needed as inputs,performs its calculations, and post outputs back to the domain.

For example, if the financial statements of a company are stored in thedomain, the domain would hold the baseline data like how much cash thecompany has, what its assets and liabilities are, etc. TheTransformation Component would be able to look at the data and calculateadditional values like cash flow ratios, ROI or NPV of investments, orany other calculations to quantitatively analyze the financial health ofthe company. Depending on their complexity, these calculations could beperformed by pre-existing spreadsheets that a client has already spentconsiderable time developing.

Remediation Component—The Remediation Component is an expert system thatfacilitates integration of intelligent feedback into BusSimapplications. It has the following features: Ability to compose highquality text feedback; Ability to compose multimedia feedback thatincludes video and/or audio; Ability to include reference material infeedback such as Authorware pages or Web Pages and Ability to activelymanipulate the users deliverables to highlight or even fix users'errors. A proven remediation theory embedded in its feedback compositionalgorithm allows integration of digital assets into the Remediation of atraining or IPS application. The Remediation model consists of threeprimary objects: Concepts; Coach Topics and Coach Items. Concepts areobjects that represent real-world concepts that the user will be facedwith in the interface. Concepts can be broken into sub-concepts,creating a hierarchical tree of concepts. This tree can be arbitrarilydeep and wide to support rich concept modeling. Concepts can also own anarbitrary number of Coach Topics. Coach Topics are objects thatrepresent a discussion topic that may be appropriate for a concept.Coach Topics can own an arbitrary number of Coach Items. Coach Items areitems of feedback that may include text, audio, video, URL's, or updatesto the Domain Model. Coach Items are owned by Coach Topics and areassembled by the Remediation Component algorithm.

Workbenches—The BusSim Toolset also includes a set of workbenches thatare used by instructional designers to design and build BusSimapplications. A workbench is a tool that facilitates visual editing ortesting of the data that the BusSim Components use for determining anapplication's run-time behavior. The BusSim Toolset includes thefollowing workbenches: Knowledge Workbench—The Knowledge Workbench is atool for the creation of domain, analysis and feedback data that is usedby the BusSim Components. It has the following features: Allows thedesigner to ‘paint’ knowledge in a drag-and-drop interface; Knowledge isrepresented visually for easy communication among designers; Theinterface is intelligent, allowing designers to only paint validinteractions; Designer's Task creations are stored in a centralrepository; The workbench supports check-in/check-out for exclusiveediting of a task; Supports LAN-based or untethered editing;Automatically generates documentation of the designs; and it Generatesthe data files that drive the behavior of the components. SimulatedStudent Test Workbench—The Simulated Student Test Workbench is a toolfor the creation of data that simulates student's actions for testingBusSim Component behaviors. It has the following features: The TestBench generates a simulated application interface based on the DomainModel; The designer manipulates the objects in the Domain Model tosimulate student activity; The designer can invoke the components toexperience the interactions the student will experience in production;and The designer can fully test the interaction behavior prior todevelopment of the application interface. Regression Test Workbench—TheRegression Test Workbench is a tool for replaying and testing of studentsessions to aid debugging. It has the following features: Each studentsubmission can be individually replayed through the components; Anarbitrary number of student submissions from the same session can bereplayed in succession; Entire student sessions can be replayed in batchinstantly; The interaction results of the student are juxtaposed withthe results of the regression test for comparison.

Development Cycle Activities

The design phase of a BusSim application is streamlined by the use ofthe Knowledge Workbench. The Knowledge Workbench is a visual editor forconfiguring the objects of the component engines to control theirruntime behavior. The components are based on proven algorithms thatcapture and implement best practices and provide a conceptual frameworkand methodology for instructional design. In conceptual design, theworkbench allows the designer to paint a model of the hierarchy ofConcepts that the student will need to master in the activity. Thishelps the designer organize the content in a logical way. The visualrepresentation of the Concepts helps to communicate ideas to otherdesigners for review. The consistent look and feel of the workbench alsocontributes to a streamlined Quality Assurance process. In addition,standard documentation can be automatically generated for the entiredesign. As the design phase progresses, the designer adds more detail tothe design of the Concept hierarchy by painting in Coach Topics that thestudent may need feedback on. The designer can associate multiplefeedback topics with each Concept. The designer also characterizes eachtopic as being Praise, Polish, Focus, Redirect or one of several othertypes of feedback that are consistent with a proven remediationmethodology. The designer can then fill each topic with text, video warstories, Web page links, Authorware links, or any other media objectthat can be delivered to the student as part of the feedback topic.

The toolset greatly reduces effort during functionality testing. The keydriver of the effort reduction is that the components can automaticallytrack the actions of the tester without the need to add code support inthe application. Whenever the tester takes an action in the interface,it is reported to the domain model. From there it can be tracked in adatabase. Testers no longer need to write down their actions for use indebugging; they are automatically written to disk. There is also afeature for attaching comments to a tester's actions. When unexpectedbehavior is encountered, the tester can hit a control key sequence thatpops up a dialog to record a description of the errant behavior. Duringthe Execution Phase, the components are deployed to the student'splatform. They provide simulated team member and feedback functionalitywith sub-second response time and error-free operation. If the clientdesires it, student tracking mechanisms can be deployed at runtime forevaluation and administration of students. This also enables theisolation of any defects that may have made it to production.

Scenarios for Using the Business Simulation Toolset

A good way to gain a better appreciation for how the BusSim Toolset canvastly improve the BusSim development effort is to walk throughscenarios of how the tools would be used throughout the developmentlifecycle of a particular task in a BusSim application. For thispurpose, we'll assume that the goal of the student in a specific task isto journalize invoice transactions, and that this task is within thebroader context of learning the fundamentals of financial accounting. Acursory description of the task from the student's perspective will helpset the context for the scenarios. Following the description are fivescenarios which describe various activities in the development of thistask. The figure below shows a screen shot of the task interface. FIG. 7illustrates the use of a toolbar to navigate and access applicationlevel features in accordance with a preferred embodiment. A student usesa toolbar to navigate and also to access some of the application-levelfeatures of the application. The toolbar is the inverted L-shaped objectacross the top and left of the interface. The top section of the toolbarallows the user to navigate to tasks within the current activity. Theleft section of the toolbar allows the student to access other featuresof the application, including feedback. The student can have hisdeliverables analyzed and receive feedback by clicking on the Teambutton.

In this task, the student must journalize twenty-two invoices and othersource documents to record the flow of budget dollars between internalaccounts. (Note: “Journalizing”, or “Journalization”, is the process ofrecording journal entries in a general ledger from invoices or othersource documents during an accounting period. The process entailscreating debit and balancing credit entries for each document. At thecompletion of this process, the general ledger records are used tocreate a trial balance and subsequent financial reports.) In accordancewith a preferred embodiment, an Intelligent Coaching Agent Tool (ICAT)was developed to standardize and simplify the creation and delivery offeedback in a highly complex and open-ended environment. Feedback from acoach or tutor is instrumental in guiding the learner through anapplication. Moreover, by diagnosing trouble areas and recommendingspecific actions based on predicted student understanding of the domainstudent comprehension of key concepts is increased. By writing rules andfeedback that correspond to a proven feedback strategy, consistentfeedback is delivered throughout the application, regardless of theinteraction type or of the specific designer/developer creating thefeedback. The ICAT is packaged with a user-friendly workbench, so thatit may be reused to increase productivity on projects requiring asimilar rule-based data engine and repository.

Definition of ICAT in Accordance with a Preferred Embodiment

The Intelligent Coaching Agent Tool (ICAT) is a suite of tools—adatabase and a Dynamic Link Library (DLL) run-time engine—used bydesigners to create and execute just-in-time feedback of Goal Basedtraining. Designers write feedback and rules in the development tools.Once the feedback is set, the run-time engine monitors user actions,fires rules and composes feedback which describes the businessdeliverable. The remediation model used within ICAT dynamically composesthe most appropriate feedback to deliver to a student based on student'sprevious responses. The ICAT model is based on a theory of feedbackwhich has been proven effective by pilot results and informalinterviews. The model is embodied in the object model and algorithms ofthe ICAT. Because the model is built into the tools, all feedbackcreated with the tool will conform to the model. ICAT plays two roles instudent training. First, the ICAT is a teaching system, helping studentsto fully comprehend and apply information. Second, ICAT is a gatekeeper,ensuring that each student has mastered the material before moving on toadditional information. ICAT is a self contained module, separate fromthe application. Separating the ICAT from the application allows otherprojects to use the ICAT and allows designers to test feedback beforethe application is complete. The ICAT Module is built on six processeswhich allow a student to interact effectively with the interface tocompose and deliver the appropriate feedback for a student's mistakes.ICAT development methodology is a seven step methodology for creatingfeedback. The methodology contains specific steps, general guidelinesand lessons learned from the field. Using the methodology increases theeffectiveness of the feedback to meet the educational requirements ofthe course. The processes each contain a knowledge model and somecontain algorithms. Each process has specific knowledge architected intoits design to enhance remediation and teaching. There is a suite oftesting tools for the ICAT. These tools allow designers and developerstest all of their feedback and rules. In addition, the utilities letdesigners capture real time activities of students as they go throughthe course. The tools and run-time engine in accordance with a preferredembodiment include expert knowledge of remediation. These objectsinclude logic that analyzes a student's work to identify problem areasand deliver focused feedback. The designers need only instantiate theobjects to put the tools to work. Embodying expert knowledge in thetools and engine ensures that each section of a course has the sameeffective feedback structure in place. A file structure in accordancewith a preferred embodiment provides a standard system environment forall applications in accordance with a preferred embodiment. Adevelopment directory holds a plurality of sub-directories. The contentin the documentation directory is part of a separate installation fromthe architecture. This is due to the size of the documentationdirectory. It does not require any support files, thus it may be placedon a LAN or on individual computers. When the architecture is installedin accordance with a preferred embodiment, the development directory hasan _Arch, _Tools, _Utilities, Documentation, QED, and XDefaultdevelopment directory. Each folder has its own directory structure thatis inter-linked with the other directories. This structure must bemaintained to assure consistency and compatibility between projects toclarify project differences, and architecture updates.

The _Arch directory stores many of the most common parts of the systemarchitecture. These files generally do not change and can be reused inany area of the project. If there is common visual basic code forapplications that will continuously be used in other applications, thefiles will be housed in a folder in this directory. The sub-directoriesin the _Arch directory are broken into certain objects of the mainproject. Object in this case refers to parts of a project that arecommonly referred to within the project. For example, modules andclasses are defined here, and the directory is analogous to a library offunctions, APIs, etc. . . . that do not change. For example the IcaObjdirectory stores code for the Intelligent Coaching Agent (ICA). TheInBoxObj directory stores code for the InBox part of the project and soon. The file structure uses some primary object references as filedirectories. For example, the IcaObj directory is a component thatcontains primary objects for the ICA such as functional forms, modulesand classes. The BrowserObj directory contains modules, classes andforms related to the browser functionality in the architecture. TheHTMLGlossary directory contains code that is used for the HTML referenceand glossary component of the architecture. The IcaObj directorycontains ICA functional code to be used in an application. This code isinstantiated and enhanced in accordance with a preferred embodiment. TheInBoxObj directory contains code pertaining to the inbox functionalityused within the architecture. Specifically, there are two majorcomponents in this architecture directory. There is a new .ocx controlthat was created to provide functionality for an inbox in theapplication. There is also code that provides support for a legacy inboxapplication. The PracticeObj directory contains code for the topicscomponent of the architecture. The topics component can be implementedwith the HTMLGlossary component as well. The QmediaObj directorycontains the components that are media related. An example is theQVIDctrl.cls. The QVIDctrl is the code that creates the links betweenQVID files in an application and the system in accordance with apreferred embodiment. The SimObj directory contains the SimulationEngine, a component of the application that notifies the tutor of inputsand outputs using a spreadsheet to facilitate communication. TheStaticObj directory holds any component that the application will usestatically from the rest of the application. For example, the login formis kept in this folder and is used as a static object in accordance witha preferred embodiment. The SysDynObj directory contains the code thatallows the Systems Dynamics Engine (Powersim) to pass values to theSimulation Engine and return the values to the tutor. The VBObjdirectory contains common Visual Basic objects used in applications. Forexample the NowWhat, Visual Basic Reference forms, and specific messagebox components are stored in this folder. The _Tools directory containstwo main directories. They represent the two most used tools inaccordance with a preferred embodiment. The two directories provide thecode for the tools themselves. The reason for providing the code forthese tools is to allow a developer to enhance certain parts of thetools to extend their ability. This is important for the current projectdevelopment and also for the growth of the tools. The Icautils directorycontains a data, database, default, graphics, icadoc, and testdatadirectory. The purpose of all of these directories is to provide asecondary working directory for a developer to keep their testingenvironment of enhanced Icautils applications separate from the projectapplication. It is built as a testbed for the tool only. No applicationspecific work should be done here. The purpose of each of thesedirectories will be explained in more depth in the project directorysection. The TestData folder is unique to the _Tools/ICAUtils directory.It contains test data for the regression bench among others componentsin ICAUtils.

The Utilities directory holds the available utilities that a BusinessSimulation project requires for optimal results. This is a repositoryfor code and executable utilities that developers and designers mayutilize and enhance in accordance with a preferred embodiment. Most ofthe utilities are small applications or tools that can be used in theproduction of simulations which comprise an executable and code to gowith it for any enhancements or changes to the utility. If new utilitiesare created on a project or existing utilities are enhanced, it isimportant to notify the managers or developers in charge of keepingtrack of the Business Simulation assets. Any enhancements, changes oradditions to the Business Simulation technology assets are important forfuture and existing projects.

In the ICAT model of feedback, there are four levels of severity oferror and four corresponding levels of feedback. The tutor goes throughthe student's work, identifies the severity of the error and thenprovides the corresponding level of feedback.

Educational Categories of Feedback ERROR FEEDBACK Error Descrip-Feedback Type tion Type Description None No errors Praise Confirmationthat the student exist. The student's completed the task correctly. workis perfect. Example: Great. You have journalized all accounts correctly.I am happy to see you recognized we are paying for most of our bills “onaccount”. Syntactic There may be Polish Tells the student the spellingmistakes or specific actions he did incorrectly, other syntactic andpossibly correct them for him. errors. As a Example: designer, you Thereare one or two errors should be confident in your work. It looks likeyou that the student will misclassified the purchase of the fax havemastered the as a cash purchase when it is really a material at thispurchase on account. point. Local A Focus Focus the student on thisparagraph of a area of his work. Point out that he paper is missing ordoes not understand at least one the student has major concept. made anumber of Example: mistakes all in one Looking over your work, I area.The student see that you do not understand the clearly does not conceptof “on account”. Why don't understand this you review that concept andreview area. your work for errors. Global The Redirect Restate the goalof the student has written activity and tell the student to review onthe wrong main concepts and retry the activity. subject or there are“There are lots of mistakes mistakes all over the throughout your work.You need to student's work think about what type of transaction eachsource document represents before journalizing it.”

Returning to the analogy of helping someone write a paper, if thestudent writes on the wrong subject, this as a global error requiringredirect feedback. If the student returns with the paper rewritten, butwith many errors in one area of the paper, focus feedback is needed.With all of those errors fixed and only spelling mistakes—syntacticmistakes—polish feedback is needed. When all syntactic mistakes werecorrected, the tutor would return praise and restate why the student hadwritten the correct paper. Focusing on the educational components ofcompleting a task is not enough. As any teacher knows, student willoften try and cheat their way through a task. Students may do no workand hope the teacher does not notice or the student may only do minorchanges in hope of a hint or part of the answer. To accommodate theseadministrative functions, there are three additional administrativecategories of feedback. The administrative and the educationalcategories of feedback account for every piece of feedback a designercan write and a student can receive. To provide a better understandingof how the feedback works together, an example is provided below.

FIG. 8 is a GBS display in accordance with a preferred embodiment. Theupper right area of the screen shows the account list. There are fourtypes of accounts: Assets, Liabilities & Equity, Revenues, and Expenses.The user clicks on one of the tabs to show the accounts of thecorresponding type. The student journalizes a transaction by dragging anaccount from the account list onto the journal entry Debits or Credits.The student then enters the dollar amounts to debit or credit eachaccount in the entry. In the interface, as in real life, the student canhave multi-legged journal entries (i.e., debiting or crediting multipleaccounts). A Toolbar 1200 and the first transaction of this Task 1210appear prominently on the display. The student can move forward and backthrough the stack of transactions. For each transaction, the studentmust identify which accounts to debit and which to credit. When thestudent is done, he clicks the Team button. FIG. 9 is a feedback displayin accordance with a preferred embodiment. The student may attempt tooutsmart the system by submitting without doing anything. The ICATsystem identifies that the student has not done a substantial amount ofwork and returns the administrative feedback depicted in FIG. 9. Thefeedback points out that nothing has been done, but it also states thatif the student does some work, the tutor will focus on the first fewjournal entries. FIG. 10 illustrates a journal entry simulation inaccordance with a preferred embodiment. FIG. 11 illustrates a simulatedBell Phone Bill journal entry in accordance with a preferred embodiment.The journal entry is accomplished by debiting Utilities Expenses andCrediting Cash for $700 each. FIG. 12 illustrates a feedback display inaccordance with a preferred embodiment. After attempting to journalizethe first three transactions, the student submits his work and receivesthe feedback depicted in FIG. 12. The feedback starts by focusing thestudent on the area of work being evaluated. The ICAT states that it isonly looking at the first three journal entries. The feedback statesthat the first two entries are completely wrong, but the third is close.If the student had made large mistakes on each of the first threetransactions, then the ICAT may have given redirect feedback, thinking aglobal error occurred. The third bullet point also highlights howspecific the feedback can become, identifying near misses.

Design Scenario—This Scenario illustrates how the tools are used tosupport conceptual and detailed design of a BusSim application. FIG. 13illustrates the steps of the first scenario in accordance with apreferred embodiment. The designer has gathered requirements anddetermined that to support the client's learning objectives, a task isrequired that teaches journalization skills. The designer begins thedesign first by learning about journalization herself, and then by usingthe Knowledge Workbench to sketch a hierarchy of the concepts she wantthe student to learn. At the most general level, she creates a rootconcept of ‘Journalization’. She refines this by defining sub-conceptsof ‘Cash related transactions’, ‘Expense related Transactions’, and‘Expense on account transactions’. These are each further refined towhatever level of depth is required to support the quality of thelearning and the fidelity of the simulation. The designer then designsthe journalization interface. Since a great way to learn is by doing,she decides that the student should be asked to Journalize a set oftransactions. She comes up with a set of twenty-two documents thattypify those a finance professional might see on the job. They includethe gamut of Asset, Expense, Liability and Equity, and Revenuetransactions. Also included are some documents that are not supposed tobe entered in the journal. These ‘Distracters’ are included becausesometimes errant documents occur in real life. The designer then usesthe Domain Model features in the Knowledge Workbench to paint a Journal.An entity is created in the Domain Model to represent each transactionand each source document. Based on the twenty-two documents that thedesigner chose, she can anticipate errors that the student might make.For these errors, she creates topics of feedback and populates them withtext. She also creates topics of feedback to tell the student when theyhave succeeded. Feedback Topics are created to handle a variety ofsituations that the student may cause.

The next step is to create profiles that will trigger topics in theconcept tree (this task is not computational in nature, so theTransformation Component does not need to be configured). A profileresolves to true when its conditions are met by the student's work. Eachprofile that resolves to true triggers a topic. To do some preliminarytesting on the design, the designer invokes the Student Simulator TestWorkbench. The designer can manipulate the Domain Model as if she werethe student working in the interface. She drags accounts around todifferent transactions, indicating how she would like them journalized.She also enters the dollar amounts that she would like to debit orcredit each account. She submits her actions to the component engines tosee the feedback the student would get if he had performed the activityin the same way. All of this occurs in the test bench without anapplication interface. The last step in this phase is low-fi usertesting. A test student interacts with a Powerpoint slide or bitmap ofthe proposed application interface for the Journalization Task. Afacilitator mimics his actions in the test bench and tells him what thefeedback would be. This simplifies low-fi user testing and helps thedesigner to identify usability issues earlier in the design when theyare much cheaper to resolve.

FIGS. 14 and 15 illustrate the steps associated with a build scenario inaccordance with a preferred embodiment. The instructional designercompletes the initial interaction and interface designs as seen in theprevious Scenario. After low-fi user testing, the Build Phase begins.Graphic artists use the designs to create the bitmaps that will make upthe interface. These include bitmaps for the buttons, tabs, andtransactions, as well as all the other screen widgets. The developerbuilds the interface using the bitmaps and adds the functionality thatnotifies the Domain Model of student actions. Standard event-drivenprogramming techniques are used to create code that will react to eventsin the interface during application execution and pass the appropriateinformation to the Domain Model. The developer does not need to have anydeep knowledge about the content because she does not have to build anylogic to support analysis of the student actions or feedback. Thedeveloper also codes the logic to rebuild the interface based on changesto the domain model. A few passes through these steps will typically berequired to get the application communicating correctly with thecomponents. The debug utilities and Regression Test Workbench streamlinethe process. After the application interface and component communicationare functioning as designed, the task is migrated to Usability testing.

The Test Scenario demonstrates the cycle that the team goes through totest the application. It specifically addresses usability testing, butit is easy to see how the tools also benefit functional and cognitiontesting. Again, we will use the Journalization Task as an example. FIG.16 illustrates a test scenario in accordance with a preferredembodiment. The test students work through the journalization activity.One of the students has made it over half way through the task and hasjust attempted to journalize the sixteenth transaction. The studentsubmits to the Financial Coach, but the feedback comes back blank. Thestudent notifies the facilitator who right-clicks on the FinancialCoach's face in the feedback window. A dialog pops up that shows this isthe twenty-seventh submission and shows some other details about thesubmission. The facilitator (or even the student in recent efforts)enters a text description of the problem, and fills out some otherfields to indicate the nature and severity of the problem. All thestudent's work and the feedback they got for the twenty-sevensubmissions is posted to the User Acceptance Test (UAT) archivedatabase. The instructional designer can review all the studenthistories in the UAT database and retrieve the session where the studentin question attempted the Journalization Task. The designer thenrecreates the problem by replaying the student's twenty-sevensubmissions through the component engines using the Regression TestWorkbench. The designer can then browse through each submission that thestudent made and view the work that the student did on the submission,the feedback the student got, and the facilitator comments, if any. Nowthe designer can use the debugging tools to determine the source of theproblem. In a few minutes, she is able to determine that additionalprofiles and topics are needed to address the specific combinations ofmistakes the student made. She uses the Knowledge Workbench to designthe new profiles and topics. She also adds a placeholder and a scriptfor a video war story that supports the learning under thesecircumstances. The designer saves the new design of the task and rerunsthe Regression Test Workbench on the student's session with the new taskdesign. After she is satisfied that the new profiles, topics, and warstories are giving the desired coverage, she ships the new task designfile to user testing and it's rolled out to all of the users.

Execution Scenario: Student Administration—FIG. 17 illustrates how thetool suite supports student administration in accordance with apreferred embodiment. When a student first enters a course she performsa pre-test of his financial skills and fills out an information sheetabout his job role, level, etc. This information is reported to theDomain Model. The Profiling Component analyzes the pre-test, informationsheet, and any other data to determine the specific learning needs ofthis student. A curriculum is dynamically configured from the TaskLibrary for this student. The application configures its mainnavigational interface (if the app has one) to indicate that thisstudent needs to learn Journalization, among other things. As thestudent progresses through the course, his performance indicates thathis proficiency is growing more rapidly in some areas than in others.Based on this finding, his curriculum is altered to give him additionalTasks that will help him master the content he is having trouble with.Also, Tasks may be removed where he has demonstrated proficiency. Whilethe student is performing the work in the Tasks, every action he takes,the feedback he gets, and any other indicators of performance aretracked in the Student Tracking Database. Periodically, part or all ofthe tracked data are transmitted to a central location. The data can beused to verify that the student completed all of the work, and it can befurther analyzed to measure his degree of mastery of the content.

Execution Scenario: Student Interaction—FIG. 18 illustrates a suite tosupport a student interaction in accordance with a preferred embodiment.In this task the student is trying to journalize invoices. He sees achart of accounts, an invoice, and the journal entry for each invoice.He journalizes a transaction by dragging and dropping an account fromthe chart of accounts onto the ‘Debits’ or the ‘Credits’ line of thejournal entry and entering the dollar amount of the debit or credit. Hedoes this for each transaction. As the student interacts with theinterface, all actions are reported to and recorded in the Domain Model.The Domain Model has a meta-model describing a transaction, its data,and what information a journal entry contains. The actions of thestudent populates the entities in the domain model with the appropriateinformation. When the student is ready, he submits the work to asimulated team member for review. This submission triggers theAnalysis-Interpretation cycle. The Transformation Component is invokedand performs additional calculations on the data in the Domain Model,perhaps determining that Debits and Credits are unbalanced for a givenjournal entry. The Profiling Component can then perform rule-basedpattern matching on the Domain Model, examining both the student actionsand results of any Transformation Component analysis. Some of theprofiles fire as they identify the mistakes and correct answers thestudent has given. Any profiles that fire activate topics in theRemediation Component. After the Profiling Component completes, theRemediation Component is invoked. The remediation algorithm searches theactive topics in the tree of concepts to determine the best set oftopics to deliver. This set may contain text, video, audio, URLs, evenactions that manipulate the Domain Model. It is then assembled intoprose-like paragraphs of text and media and presented to the student.The text feedback helps the student localize his journalization errorsand understand why they are wrong and what is needed to correct themistakes. The student is presented with the opportunity to view a videowar story about the tax and legal consequences that arise from incorrectjournalization. He is also presented with links to the referencematerials that describe the fundamentals of journalization. TheAnalysis-Interpretation cycle ends when any coach items that result inupdates to the Domain Model have been posted and the interface isredrawn to represent the new domain data. In this case, the designerchose to highlight with a red check the transactions that the studentjournalized incorrectly.

The Functional Definition of the ICAT

This section describes the feedback processes in accordance with apreferred embodiment. For each process, there is a definition of theprocess and a high-level description of the knowledge model. Thisdefinition is intended to give the reader a baseline understanding ofsome of the key components/objects in the model, so that he can proceedwith the remaining sections of this paper. Refer to the DetailedComponents of the ICAT for a more detailed description of each of thecomponents within each knowledge model. To gain a general understandingof the ICAT, read only the general descriptions. To understand the ICATdeeply, read this section and the detailed component section regardingknowledge models and algorithms. These processes and algorithms embodythe feedback model in the ICAT. There are six main processes in theICAT, described below and in more detail on the following pages.

FIG. 19 illustrates the remediation process in accordance with apreferred embodiment. Remediation starts as students interact with theapplication's interface (process 1901). As the student tries to completethe business deliverable, the application sends messages to ICAT abouteach action taken (process 1902). When the student is done and submitswork for review, the ICAT compares how the student completed theactivity with how the designer stated the activity should be completed(this is called domain knowledge). From this comparison, the ICAT gets acount of how many items are right, wrong or irrelevant (process 1903).With the count complete, the ICAT tries to fire all rules (process1904). Any rules which fire activate a coach topic (process 1905). Thefeedback algorithm selects pieces of feedback to show and composes theminto coherent paragraphs of text (process 1906). Finally, as part ofcreating feedback text paragraphs, the ICAT replaces all variables inthe feedback with specifics from the student's work. This gives thefeedback even more specificity, so that it is truly customized to eachstudent's actions.

Knowledge Model—Interface Objects In any GBS Task, the student mustmanipulate controls on the application interface to complete therequired deliverables. FIG. 20 illustrates the objects for thejournalization task in accordance with a preferred embodiment. Thefollowing abstract objects are used to model all the various types ofinterface interactions. A SourceItem is an object the student uses tocomplete a task. In the journalization example, the student makes adebit and credit for each transaction. The student has a finite set ofaccounts with which to respond for each transaction. Each account thatappears in the interface has a corresponding SourceItem object. In otherwords, the items the student can manipulate to complete the task(account names) are called SourceItems. A Source is an object thatgroups a set of SourceItem objects together. Source objects have aOne-To-Many relationship with SourceItem objects. In the journalizationexample, there are four types of accounts: Assets, Liabilities andEquity, Revenues, and Expenses. Each Account is of one and only one ofthese types and thus appears only under the appropriate tab. For each ofthe Account type tabs, there is a corresponding Source Object. A Targetis a fixed place where students place SourceItems to complete a task. Inthe journalization example, the student places accounts on two possibletargets: debits and credits. The top two lines of the journal entrycontrol are Debit targets and the bottom two lines are Credit targets.These two targets are specific to the twelfth transaction. A TargetPageis an object that groups a set of Target objects together. TargetPageobjects have a One-To-Many relationship with Target objects (just likethe Source to SourceItem relationship). In the journalization example,there is one journal entry for each of the twenty-two transactions. Foreach journal entry there is a corresponding TargetPage object thatcontains the Debits Target and Credits Target for that journal entry.

As the student manipulates the application interface, each action isreported to the ICAT. In order to tell the ICAT what actions were taken,the application calls to a database and asks for a specific interfacecontrol's ID. When the application has the ID of the target control andthe SourceItem control, the application notifies the ICAT about theTarget to SourceItem mapping. In other words, every time a studentmanipulates a source item and associates it with a target (e.g.,dragging an account name to a debit line in the journal), the useraction is recorded as a mapping of the source item to the target. Thismapping is called a UserSourceItemTarget. FIG. 21 illustrates themapping of a source item to a target item in accordance with a preferredembodiment. When the student is ready, he submits his work to one of thesimulated team members by clicking on the team member's icon. When theICAT receives the student's work, it calculates how much of the work iscorrect by concept. Concepts in our journalization activity will includeDebits, Credits, Asset Accounts, etc. For each of these concepts, theICAT will review all student actions and determine how many of thestudent actions were correct. In order for the ICAT to understand whichtargets on the interface are associated with each concept, the targetsare bundled into target groups and prioritized in a hierarchy. Once allpossible coach topics are activated, a feedback selection analyzes theactive pieces of remediation within the concept hierarchy and selectsthe most appropriate for delivery. The selected pieces of feedback arethen assembled into a cohesive paragraph of feedback and delivered tothe student. FIG. 23 illustrates a feedback selection in accordance witha preferred embodiment. After the ICAT has activated CoachTopics viaRule firings, the Feedback Selection Algorithm is used to determine themost appropriate set of CoachItems (specific pieces of feedback textassociated with a CoachTopic) to deliver. The Algorithm accomplishesthis by analyzing the concept hierarchy (TargetGroup tree), the activeCoachTopics, and the usage history of the CoachItems. FIG. 24 is aflowchart of the feedback logic in accordance with a preferredembodiment. There are five main areas to the feedback logic whichexecute sequentially as listed below. First, the algorithm looks throughthe target groups and looks for the top-most target group with an activecoach topic in it. Second, the algorithm then looks to see if thattop-most coach item is praise feedback. If it is praise feedback, thenthe student has correctly completed the business deliverable and theICAT will stop and return that coach item. Third, if the feedback is notPraise, then the ICAT will look to see if it is redirect, polish,mastermind or incomplete-stop. If it is any of these, then the algorithmwill stop and return that feedback to the user. Fourth, if the feedbackis focus, then the algorithm looks to the children target groups andgroups any active feedback in these target groups with the focus groupheader. Fifth, once the feedback has been gathered, then thesubstitution language is run which replaces substitution variables withthe proper names. Once the ICAT has chosen the pieces of feedback toreturn, the feedback pieces are assembled into a paragraph. With theparagraph assembled, the ICAT goes through and replaces all variables.There are specific variables for SourceItems and Targets. Variables givefeedback specificity. The feedback can point out which wrong SourceItemswere placed on which Targets. It also provides hints by providing one ortwo SourceItems which are mapped to the Target.

The Steps Involved in Creating Feedback in Accordance with a PreferredEmbodiment

The goal of feedback is to help a student complete a businessdeliverable. The tutor needs to identify which concepts the studentunderstands and which he does not. The tutor needs to tell the studentabout his problems and help him understand the concepts. There are sevenmajor steps involved in developing feedback for an application. First,creating a strategy—The designer defines what the student should know.Second, limit errors through interface—The designer determines if theinterface will identify some low level mistakes. Third, creating atarget group hierarchy—The designer represents that knowledge in thetutor. Fourth, sequencing the target group hierarchy—The designer tellsthe tutor which concepts should be diagnosed first. Fifth, writingfeedback—The designer writes feedback which tells the student how he didand what to do next. Sixth, writing Levels of Feedback—The designerwrites different levels of feedback in case the student makes the samemistake more than once. Seventh, writing rules—The designer definespatterns which fire the feedback.

A feedback strategy is a loose set of questions which guide the designeras he creates rules and feedback. The strategy describes what thestudent should learn, how he will try and create the businessdeliverable and how an expert completes the deliverable. The goal of theapplication should be for the student to transition from the novicemodel to the expert model. What should the student know after using theapplication? The first task a designer needs to complete is to defineexactly what knowledge a student must learn by the end of theinteraction. Should the student know specific pieces of knowledge, suchas formulas? Or, should the student understand high level strategies anddetailed business processes? This knowledge is the foundation of thefeedback strategy. The tutor needs to identify if the student has usedthe knowledge correctly, or if there were mistakes. An example is thejournal task. For this activity, students need to know the purpose ofthe journalizing activity, the specific accounts to debit/credit, andhow much to debit/credit. A student's debit/credit is not correct orincorrect in isolation, but correct and incorrect in connection with thedollars debited/credited. Because there are two different types ofknowledge—accounts to debit/credit and amounts to debit/credit—thefeedback needs to identify and provide appropriate feedback for bothtypes of mistakes.

How will a novice try and complete the task? Designers should start bydefining how they believe a novice will try and complete the task. Whichareas are hard and which are easy for the student. This novice view isthe mental model a student will bring to the task and the feedbackshould help the student move to an expert view. Designers should payspecial attention to characteristic mistakes they believe the studentwill make. Designers will want to create specific feedback for thesemistakes. An example is mixing up expense accounts in the journalactivity. Because students may mix up some of these accounts, thedesigner may need to write special feedback to help clear up anyconfusion.

How does an expert complete the task? This is the expert model ofcompleting the task. The feedback should help students transition tothis understanding of the domain. When creating feedback, a designershould incorporate key features of the expert model into the praisefeedback he writes. When a student completes portion of the task,positive reinforcement should be provided which confirms to the studentthat he is doing the task correctly and can use the same process tocomplete the other tasks. These four questions are not an outline forcreating feedback, but they define what the feedback and the wholeapplication needs to accomplish. The designer should make sure that thefeedback evaluates all of the knowledge a student should learn. Inaddition, the feedback should be able to remediate any characteristicmistakes the designer feels the student will make. Finally, the designershould group feedback so that it returns feedback as if it were anexpert. With these components identified, a designer is ready to startcreating target group hierarchies. Because there are positive andnegative repercussions, designers need to select the when to remediatethrough the interface carefully. The criteria for making the decision isif the mistake is a low level data entry mistake or a high levelintellectual mistake. If the mistake is a low level mistake, such asmiss-typing data, it may be appropriate to remediate via the interface.If the designer decides to have the interface point out the mistakes, itshould look as if the system generated the message. System generatedmessages are mechanical checks, requiring no complex reasoning. Incontrast, complex reasoning, such as why a student chose a certain typeof account to credit or debit should be remediated through the ICAT.

System messages—It is very important that the student know what type ofremediation he is going to get from each source of information.Interface based remediation should look and feel like system messages.They should use a different interface from the ICAT remediation andshould have a different feel. In the journalization task describedthroughout this paper, there is a system message which states “Creditsdo not equal debits.” This message is delivered through a differentinterface and the blunt short sentence is unlike all other remediation.The motivation for this is that low level data entry mistakes do notshow misunderstanding but instead sloppy work. Sloppy-work mistakes donot require a great deal of reasoning about why they occurred instead,they simply need to be identified. High-level reasoning mistakes,however, do require a great deal of reasoning about why they occurred,and the ICAT provides tools, such as target groups, to help with complexreasoning. Target group hierarchies allow designers to group mistakesand concepts together and ensure that they are remediated at the mostappropriate time (i.e., Hard concepts will be remediated before easyconcepts). Timing and other types of human-like remediation require theICAT; other low-level mistakes which do not require much reasoninginclude: Incomplete—If the task requires a number of inputs, theinterface can check that they have all been entered before allowing thestudent to proceed. By catching empty fields early in the process, thestudent may be saved the frustration of having to look through eachentry to try and find the empty one. Empty—A simple check for the systemis to look and see if anything has been selected or entered. If nothinghas been selected, it may be appropriate for the system to generate amessage stating “You must complete X before proceeding”. Numbers notmatching—Another quick check is matching numbers. As in thejournalization activity, is often useful to put a quick interface checkin place to make sure numbers which must match do. Small data entrymistakes are often better remediated at the interface level than at thetutor or coach level (when they are not critical to the learningobjectives of the course). There are two main issues which must beremembered when using the interface to remediate errors. First, makesure the interface is remediating low level data entry errors. Second,make sure the feedback looks and feels different from the ICAT feedback.The interface feedback should look and feel like it is generated fromthe system while the ICAT feedback must look as if it were generatedfrom an intelligent coach who is watching over the student as he works.

Creating the Target Group Hierarchy—Target groups are sets of targetswhich are evaluated as one. Returning to the severity principle of thefeedback theory, it is clear that the tutor needs to identify how muchof the activity the student does not understand. Is it a global problemand the student does not understand anything about the activity? Or, isit a local problem and the student simply is confused over one concept?Using the feedback algorithm described earlier, the tutor will returnthe highest target group in which there is feedback. This algorithmrequires that the designer start with large target groups and makesub-groups which are children of the larger groups. The ICAT allowsstudents to group targets in more than one category. Therefore a debittarget for transaction thirteen can be in a target group for transactionthirteen entries as well as a target group about debits and a targetgroup which includes all source documents. Target should be grouped withfour key ideas in mind. Target groups are grouped according to: Conceptstaught; Interface constraints; Avoidance of information overload andPositive reinforcement.

The most important issue when creating target groups is to create themalong the concepts students need to know to achieve the goal. Groupingtargets into groups which are analogous to the concepts a student needsto know, allows the tutor to review the concepts and see which conceptsconfuse the student. As a first step, a designer should identify in anunstructured manner all of the concepts in the domain. This first passwill be a large list which includes concepts at a variety ofgranularities, from small specific concepts to broad general concepts.These concepts are most likely directly related to the learningobjectives of the course. With all of the concepts defined, designersneed to identify all of the targets which are in each target group. Sometargets will be in more than one target group. When a target is in morethan one target group, it means that there is some type of relationshipsuch as a child relationship or a part to whole relationship. The pointis not to create a structured list of concepts but a comprehensive list.Structuring them into a hierarchy will be the second step of theprocess.

* Notes: Loads from Database or Document based on values

* of m_StorageTypeTask and m_StorageTypeStudent

*

******************************************

*/

extern “C”

{

-   -   long _export WINAPI TuResumeStudent(long StudentID, long TaskID,        int fromSubmissionSeqID); // Resumes a Student's work for the        Task at the specified Submission

}

extern “C”

{

-   -   long _export WINAPI TuLoadArchivedSubmissions(long StudentID,        long TaskID, int fromSubmissionSeqID, int toSubmissionSeqID); //        Loads Archived Submissions For a Student's work in a Task

}

extern “C”

{

-   -   long _export WINAPI TuUseArchivedSubmissions(int n); // Replays        n Archived submissions for debugging

}

extern “C”

{

-   -   long _export WINAPI TuSaveCurrentStudent( ); // Saves Current        Student's work to DB

}

extern “C”

{

-   -   long _export WINAPI KillEngine(long ITaskID); // Delete all        Dynamic objects before shutdown

* Function Return

* Variables: TUT_ERR_OK

*

* Notes:

******************************************

*/

extern “C”

{

-   -   long _export WINAPI TuSetTaskDocPathName(LPCSTR fnm);

}

/*

******************************************

* Name: TuSetFeedbackFileName

* Purpose: To set path and name of file to use for holding feedback

* Input

* Parameters: LPCSTR fnm

*

* Path and name of file to use for holding feedback

* Output

* Parameters: none

*

* Function Return

* Variables: TUT_ERR_OK

*

* Notes:

******************************************

*/

extern “C”

{

-   -   long _export WINAPI TuSetFeedbackFileName(LPCSTR fnm);

}

/*

******************************************

* Name: TuSetFeedbackPrevFileName

* Purpose: To set path and name of file to use for holding previousfeedback

* Input

* Parameters: LPCSTR fnm

* Path and name of file to use for holding previous feedback

* Output

* Parameters: none

*

* Function Return

* Variables: TUT_ERR_OK

* Notes:

******************************************

*/

extern “C”

{

-   -   long _export WINAPI TuSetFeedbackPrevFileName(LPCSTR fnm);

/*

******************************************

* Name: TuSetLogFileName

* Purpose: To set path and name of file to use for full logging

* Input

* Parameters: LPCSTR fnm

* Path and name of file to use for full logging

* Output

* Parameters: none

*

* Function Return

* Variables: TUT_ERR_OK

* Notes:

******************************************

*/

extern “C”

{

-   -   long _export WINAPI TuSetLogFileName(LPCSTR fnm);

}

/*

******************************************

* Name: TuSetLogLoadFileName

* Purpose: To set path and name of file to use for load logging

* Input

* Parameters: LPCSTR fnm

* Path and name of file to use for load logging

* Output

* Parameters: none

*

* Function Return

* Variables: TUT_ERR_OK

*

* Notes:

******************************************

*/

extern “C”

{

-   -   long _export WINAPI TuSetLogLoadFileName(LPCSTR fnm);

}

/*

******************************************

* Name: TuSetLogStudentFileName

* Purpose: To set path and name of file to use for student logging

* Input

* Parameters: LPCSTR fnm

* Path and name of file to use for student logging

* Output

* Parameters: none

*

* Function Return

* Variables: TUT_ERR_OK

*

* Notes:

******************************************

*/

extern “C”

}

-   -   long _export WINAPI TuSetLogStudentFileName(LPCSTR fnm);

}

/*

******************************************

* Name: TuSetLogSubmissionFileName

* Purpose: To set path and name of file to use for submission logging

* Input

* Parameters: LPCSTR fnm

* Path and name of file to use for submission logging

* Output

* Parameters: none

*

* Function Return

* Variables: TUT_ERR_OK

*

* Notes:

******************************************

*/

extern “C”

{

-   -   long _export WINAPI TuSetLogSubmissionFileName(LPCSTR fnm);

}

/*

******************************************

* Name: TuSetLogErrFileName

* Purpose: To set path and name of file to use for error logging

* Input

* Parameters: LPCSTR fnm

* Path and name of file to use for error logging

* Output

* Parameters: none

*

* Function Return

* Variables: TUT_ERR_OK

* Notes:

******************************************

*/

extern “C”

{

-   -   long _export WINAPI TuSetLogErrFileName(LPCSTR fnm);

}

*/

******************************************

* Name: TuSetTrace

* Purpose: To turn Trace on and off

* Input

* Parameters: int TraceStatus

* TUT_TRACE_ON :Turn Trace On

* TUT_TRACE_OFF :Turn Trace Off

* Output

* Parameters: none

*

* Function Return

* Variables: Previous Trace Status Value

* TUT_TRACE_ON

* TUT_TRACE_OFF

*

* TUT_ERR_INVALID_TRACE_STATUS

* Notes:

******************************************

*/

extern “C”

{

-   -   long _export WINAPI TuSetTrace(int TraceStatus);

}

/*

******************************************

* Name: TuSetTrack

* Purpose: To turn Tracking on and off. While tracking is on

* all work the user does and all feedback the user receives

* is kept. While Tracking is off only the most recent work is kept.

* Input

* Parameters: int TrackStatus

* TUT_TRACK_ON :Turn Tracking On

* TUT_TRACK_OFF :Turn Tracking Off

* Output

* Parameters: none

* Function Return

* Variables: Previous Trace Status Value

* TUT_TRACK_ON

* TUT_TRACK_OFF

*

* TUT_ERR_INVALID_TRACK_STATUS

* Notes:

******************************************

*/

extern “C”

{

-   -   long _export WINAPI TuSetTrack(int TrackStatus);

}

Simulation Engine

The idea is for the designer to model the task that he wants a studentto accomplish using an Excel spreadsheet. Then, have an algorithm orengine that reads all the significant cells of the spreadsheet andnotifies the Intelligent Coaching Agent with the appropriate information(SourceItemID, TargetID and Attribute). This way, the spreadsheet actsas a central repository for student data, contains most of thecalculations required for the task and in conjunction with the enginehandles all the communication with the ICA. The task is self containedin the spreadsheet, therefore the designers no longer need a graphicaluser interface to functionally test their designs (smart spreadsheet.Once the model and feedback for it are completely tested by designers,developers can incorporate the spreadsheet in a graphical userinterface, e.g., Visual Basic as a development platform. The simulationspreadsheet is usually invisible and populated using functions providedby the engine. It is very important that all modifications that the ICAneeds to know about go through the engine because only the engine knowshow to call the ICA. This significantly reduced the skill level requiredfrom programmers, and greatly reduced the time required to program eachtask. In addition, the end-product was less prone to bugs, because thetutor management was centralized. If there was a tutor problem, we onlyhad to check on section of code. Finally, since the simulation engineloaded the data from a spreadsheet, the chance of data inconsistencybetween the tutor and the application was nil.

FIG. 25 is a block diagram setting forth the architecture of asimulation model in accordance with a preferred embodiment. TheSimulation Object Model consists of a spreadsheet model, a spreadsheetcontrol object, a simulation engine object, a simulation database, inputobjects, output objects, list objects and path objects. The first objectin our discussion is the Spreadsheet object. The Spreadsheet is thesupport for all simulation models. A control object that is readilyintegrated with the Visual Basic development plat. The control supportsprinting and is compatible with Microsoft Excel spreadsheets. With thatin mind, designers can use the power of Excel formulas to build thesimulation. The different cells contained in the spreadsheet model canbe configured as Inputs, Outputs or Lists and belong to a simulationPath. All cells in the spreadsheet that need to be manually entered bythe designer or the student via the GBS application are represented byinput objects. Every input has the following interface:

Data Field Name Type Description InputID long Primary Key for the tableTaskID long TaskID of the task associated with the input PathID longPathID of the path associated with the input InputName string*50 Name ofthe input InputDesc string*255 Description of the input ReferenceNamestring*50 Name of the spreadsheet cell associated with the inputTutorAware boolean Whether the ICA should be notified of any changes tothe input SourceItemID long SourceItemID if input is a distinct input; 0if input is a drag drop input TargetID long TargetID of the input Rowlong Spreadsheet row number of the input → speed optimization Columnlong Spreadsheet column number of the input → speed optimizationSheetName string*50 Sheet name were the input is located → speedoptimization

This information is stored for every input in the Input table of thesimulation database (ICASim.mdb). Refer to the example below. Whendesigners construct their simulation model, they must be aware of thefact that there are 2 types of Inputs: Distinct Input & Drag & DropInput. The Distinct Input consists of a single spreadsheet cell that canbe filled by the designer at design time or by the GBS application atrun time via the simulation engine object's methods. The purpose of thecell is to provide an entry point to the simulation model. This entrypoint can be for example an answer to a question or a parameter to anequation. If the cell is TutorAware (all inputs are usually TutorAware),the ICA will be notified of any changes to the cell. When the ICA isnotified of a change two messages are in fact sent to the ICA: AnICANotifyDestroy message with the input information i.e., SourceItemID,TargetID and null as Attribute. This message is to advise the ICA toremove this information from its memory. An ICANotifyCreate message withthe input information i.e., SourceItemID, TargetID, Attribute (cellnumeric value). This message is to advise the ICA to add thisinformation to its memory. A Distinct Input never requires that a useranswer a mathematics question.

These are the steps required to configure that simulation: Define a namefor cell C2 in Excel. Here we have defined “Distinct_Input”. In the ICA,define a task that will be assigned to the simulation. Ex: a TaskID of123 is generated by the ICA. In the ICA, define a Target for the input.Ex: a TargetID of 4001 is generated by the ICA. In the ICA, define aSourceItem for the input. Ex: a SourceItemID of 1201 is generated by theICA. Associate the input to a path (refer to Path object discussion).Add the information in the Input table of the simulation enginedatabase. A record in an Input table is presented below.

InputID: 12345 TaskID: 123 PathID: 1234 InputName: Question 1 inputInputDesc: Distinct input for Question 1 ReferenceName: Distinct_inputTutorAware: True SourceItemID 1201 TargetID: 4001 Row: 2 Column: 3SheetName: Sheet1

The Row, Column and SheetName are filled in once the user clicks “RunInputs/Outputs”. The simulation engine decodes the defined name(Reference Name) that the designer entered, and populates the tableaccordingly. This is an important step. We had several occasions when adesigner would change the layout of a spreadsheet, i.e., move a definedname location, then forget to perform this step. As such, bizarre datawas being passed to the tutor; whatever data happened to reside in theold row and column. Once the configuration is completed, the designercan now utilize the ICA Utilities to test the simulation.

The drag & drop input consist of two consecutive spreadsheet cells. Bothof them have to be filled by the designer at design time or by the GBSapplication at run time via the simulation engine object's methods. Thistype of input is used usually when the user must choose one answer amonga selection of possible answers. Drag & drop inputs are alwaysTutorAware. The left most cell contains the SourceItemID of the answerpicked by the user (every possible answer needs a SourceItemID) and therightmost cell can contain a numeric value associated to that answer.You need to define a name or ReferenceName in the spreadsheet for therightmost cell. ICA will be notified of any changes to either one of thecells. When the ICA is notified of a change two messages are in factsent to the ICA: An ICANotifyDestroy message with the input informationi.e., SourceItemID before the change occurred, TargetID of the input andthe Attribute value before the change occurred. An ICANotifyCreatemessage with the input information i.e., SourceItemID after the changeoccurred, TargetID of the input and the Attribute value after the changeoccurred.

These are the steps required to configure that section of thesimulation: Define a name for cell C11 in Excel. Here we have defined“DragDrop_Input”; Let's use the same TaskID as before since Question 2is part of the same simulation as Question 1. Ex: TaskID is 123; In theICA, define a Target for the input. Ex: a TargetID of 4002 is generatedby the ICA; In the ICA, define a SourceItem for every possible answer tothe question. Ex: SourceItemIDs 1202 to 1205 are generated by the ICA;Associate the input to a path (refer to Path object discussion); and Addthe information in the Input table of the simulation engine database. Arecord in the Input table in accordance with a preferred embodiment ispresented below.

InputID: 12346 TaskID: 123 PathID: 1234 InputName: Question 2 inputInputDesc: Drag & Drop input for Question 2 ReferenceName:DragDrop_Input TutorAware: True SourceItemID 0*** TargetID: 4002 Row: 11Column: 3 SheetName: Sheet1

The list object consists of one cell identifying the list (cell #1) anda series of placeholder rows resembling drag & drop inputs (cells#1.1-1.n to cells #n.1-n.n). The list is used usually when the user mustchoose multiple elements among a selection of possible answers. Cell #1must have a uniquely defined name also called the list name. Cells #1.1to #n.1 can contain the SourceItemID of one possible answer picked bythe user (every possible answer needs a SourceItemID). The content ofthese cells must follow this format: ˜ListName˜SourceItemID. Cells #1.2to #n.2 will hold the numeric value (attribute) associated with theSourceItemID in the cell immediately to the left. Cells #1.3-1.n to#n.3-n.n are optional placeholders for data associated with the answer.KEY NOTE: When implementing a list object the designer must leave allthe cells under #n.1 to #n.n blank because this range will shift upevery time an item is removed from the list.

Every list has the following interface:

Data Field Name Type Description ListID long Primary Key for the tableTaskID long TaskID of the task associated with the list PathID longPathID of the path associated with the list ListName string*50 Name ofthe list ListDesc string*255 Description of the list ReferenceNamestring*50 Name of the spreadsheet cell associated with the listTutorAware boolean Whether the ICA should be notified of any changes tothe list TargetID long TargetID of the output TotalColumns long Totalnumber of data columns Row long Spreadsheet row number of the output →speed optimization Column long Spreadsheet column number of the output →speed optimization SheetName string*50 Sheet name were the input islocated → speed optimization

Use of a list is demonstrated by continuing our math test. The mathquestion in this example invites the user to select multiple elements toconstruct the answer. These are the steps required to configure thatsection of the simulation. FIG. 26 illustrates the steps for configuringa simulation in accordance with a preferred embodiment. Define a namefor cell C23 in Excel. Here we have defined “The_List”. Let's use thesame Task ID as before since Question 3 is part of the same simulationas Question 1 and 2. Ex: TaskID is 123. In the ICA, define a Target forthe list. Ex: a TargetID of 4006 is generated by the ICA. In the ICA,define a SourceItem for every item that could be placed in the list. Ex:the following SourceItemIDs 1209, 1210, 1211. 1212, 1213, 1214 aregenerated by the ICA. Associate the list to a path (refer to Path objectdiscussion). Add the information in the List table of the simulationengine database.

-   -   A record in the List table in accordance with a preferred        embodiment is presented in the table appearing below.

ListID: 12346 TaskID: 123 PathID: 1234 ListName: Question 3 listListDesc: List for Question 3 ReferenceName: The_List TutorAware: TrueTargetID: 4006 TotalColumns: 1 Row: 23 Column: 3 SheetName: Sheet1

All cells in the spreadsheet that are result of calculations (do notrequire any external input) can be represented by output objects. Everyoutput has an interface as outlined in the table below.

Data Field Name Type Description OutputID long Primary Key for the tableTaskID long TaskID of the task associated with the output PathID longPathID of the path associated with the output OutputName string*50 Nameof the output OutputDesc string*255 Description of the outputReferenceName string*50 Name of the spreadsheet cell associated with theoutput TutorAware boolean Whether the ICA should be notified of anychanges to the output SourceItemID long SourceItemID of the outputTargetID long TargetID of the output Row long Spreadsheet row number ofthe output → speed optimization Column long Spreadsheet column number ofthe output → speed optimization SheetName string*50 Sheet name were theinput is located → speed optimization

All this information is stored for every output in the Output table ofthe simulation database (ICASim.mdb). When designers construct theirsimulation model, they must be aware of the fact that there is only 1type of Outputs: the Distinct Output. A Distinct Output consists of oneand only one spreadsheet cell that contains a formula or a result ofcalculations. The existence of Output cells is the main reason to have asimulation model. If the cell is TutorAware, the ICA will be notified ofany changes to the cell when all outputs are processed otherwise the ICAwill be unaware of any changes. When the ICA is notified of a change twomessages are in fact sent to the ICA: An ICANotifyDestroy message withthe output information i.e., SourceItemID, TargetID and null asAttribute. This message is to advise the ICA to remove this informationfrom its memory. An ICANotifyCreate message with the output informationi.e., SourceItemID, TargetID, Attribute (cell numeric value). Thismessage is to advise the ICA to add this information to its memory. Asopposed to Distinct Inputs and Drag & Drop Inputs which notify the ICAon every change, Distinct Outputs are processed in batch just beforeasking the ICA for feedback. To notify the ICA of the total dollaramount of the items in the list. We definitely need a Distinct Outputfor that. The output will contain a sum formula. Define a name for cellC24 in Excel. Here we have defined “Distinct Output”. Let's use the sameTaskID as before since Question 3 is part of the same simulation asQuestion 1 and 2. Ex: TaskID is 123. In the ICA, define a Target for theoutput. Ex: a TargetID of 4005 is generated by the ICA. In the ICA,define a SourceItem for the output. Ex: a SourceItemID of 1215 isgenerated by the ICA. Associate the output to a path (refer to Pathobject discussion). Add the information in the Output table of thesimulation engine database.

A record in an Output table in accordance with a preferred embodiment ispresented below.

OutputID: 12347 TaskID: 123 PathID: 1234 OutputName: Question 3 outputOutputDesc: Distinct Output for Question 3 ReferenceName:Distinct_Output TutorAware: True SourceItemID 1215 TargetID: 4005 Row:24 Column: 6 SheetName: Sheet1

Paths are used to divide a simulation model into sub-Simulations meaningthat you can group certain inputs, outputs and lists together to form acoherent subset or path. Every path has the following interface:

Data Field Name Type Description PathID long Primary Key for the tableTaskID long TaskID of the task associated with the path PathNo longNumeric value associated to a path PathName string*50 Name of the pathPathDesc string*255 Description of the path

All this information is stored for every path in the path table of thesimulation database (ICASim.mdb).

The simulation engine is the interface between the model, the simulationdatabase and the Intelligent Coaching Agent. The simulation engine is ofinterest to the designer so that he can understand the mechanics of itall. But it is the developer of applications using the engine thatshould know the details of the interface (methods & properties) exposedby the engine and the associated algorithms. Once the designer hasconstructed the simulation model (Excel Spreadsheet) and configured allthe inputs, outputs & lists, he is ready to test using the test benchincluded in the ICA Utilities (refer to ICA Utilities documentation).The developer, in turn, needs to implement the calls to the simulationengine in the GBS application he's building. The following listidentifies the files that need to be included in the Visual Basicproject to use the simulation workbench

wSimEng.cls Simulation Engine class wSimEng.bas Simulation Engine module(this module was introduced only for speed purposes because all the codeshould theoretically be encapsulated in the class) wConst.basIntelligent Coaching Agent constant declaration wDeclare.bas IntelligentCoaching Agent DLL interface wlca.cls Intelligent Coaching Agent classwlca.bas Intelligent Coaching Agent module (this module was introducedonly for speed purposes because all the code should theoretically beencapsulated in the class)

To have a working simulation, a developer places code in differentstrategic areas or stages of the application. There's the Initial stagethat occurs when the form containing the simulation front-end loads.This is when the simulation model is initialized. There's theModification stages that take place when the user makes changes to thefront-end that impacts the simulation model. This is when the ICA isnotified of what's happening. There's the Feedback stage when the userrequests information on the work done so far. This is when thesimulation notifies the ICA of all output changes. Finally, there's theFinal stage when the simulation front-end unloads. This is when thesimulation is saved to disk.

The different stages of creating a simulation, including the VisualBasic code involved, are presented below. Initial stage; 1. Creating theICA & the simulation engine object: Code: Set moSimEngine=NewclassSimEngine; Set moICA=New classICA; Description: The first step inusing the simulation engine is to create an instance of the classclassSimEngine and also an instance of the class classICA. Note that theengine and ICA should be module level object “mo” variables. 2. Loadingthe simulation; Code: IRet=moSimEngine.OpenSimulation(App.Path &DIR_DATA & FILE_SIMULATION, Me.bookSimulation);IRet=moSimEngine.LoadSimulation(mIICATaskID, App.Path & DIR_DATABASE &DB_SIMULATION, 1); Description: After the object creation, theOpenSimulation and LoadSimulation methods of the simulation engineobject must be called. The OpenSimulation method reads the specifiedExcel 5.0 spreadsheet file into a spreadsheet control. TheLoadSimulation method opens the simulation database and loads intomemory a list of paths, a list of inputs, a list of outputs and a listof lists for the specific task. Every method of the simulation enginewill return 0 if it completes successfully otherwise an appropriateerror number is returned. 3. Initializing and loading the IntelligentCoaching Agent; Code:IRet=moICA.Initialize(App.Path & “\” & App.EXEName& “.ini”, App.Path & DIR_DATABASE, App.Path & DIR_ICADOC, App.Path &“\”); IRet=moICA.LoadTask(mIICATaskID, ICAStudentStartNew); Description:The simulation engine only works in conjunction with the ICA. TheInitialize method of the ICA object reads the application .ini file andsets the Tutor32.dll appropriately. The LoadTask method tells the ICA(Tutor32.dll) to load the .tut document associated to a specific task inmemory. From that point on, the ICA can receive notifications. Note: Thetut document contains all the element and feedback structure of a task.Ex: SourcePages, SourceItems, TargetPages, Targets, etc. . . . 4.Restoring the simulation; Code: <<Code to reset the simulation whenstarting over>>; <<Code to load the controls on the simulationfront-end>>; IRet=moSimEngine.RunInputs(sPaths, True);IRet=moSimEngine.RunOutputs(sPaths, True);IRet=moSimEngine.RunLists(sPaths, True); Call moICA.Submit(0); CallmoICA.SetDirtyFlag(0, False); Description: Restoring the simulationinvolves many things: clearing all the inputs and lists when the user isstarting over; loading the interface with data from the simulationmodel; invoking the RunInputs, RunOutputs and RunLists methods of thesimulation engine object in order to bring the ICA to it's originalstate; calling the Submit method of the ICA object with zero as argumentto trigger all the rules; calling the SetDirtyFlag of the ICA objectwith 0 and false as arguments in order to reset the user's session.Running inputs involves going through the list of TutorAware inputs andnotifying the ICA of the SourceItemID, TargetID and Attribute value ofevery input. Running lists involves going through the list of TutorAwarelists and notifying the ICA of the SourceItemID, TargetID and Attributevalue of every item in every list. The TargetID is unique for every itemin a list.

Running outputs involves going through the list of TutorAware outputsand notifying the ICA of the SourceItemID, TargetID and Attribute valueof every output. Modification stage 1. Reading inputs & outputs; Code:Dim sDataArray(2) as string; Dim vAttribute as variant; DimISourceItemID as long; Dim ITargetID as long;IRet=moSimEngine.ReadReference(“Distinct_Input”, vAttribute,ISourceItemID, ITargetID, sDataArray)

Description: The ReadReference method of the simulation object willreturn the attribute value of the input or output referenced by name andoptionally retrieve the SourceItemID, TargetID and related data. In thecurrent example, the attribute value, the SourceItemID, the TargetID and3 data cells will be retrieved for the input named Distinct_Input.

Description: The simulation engine object provides basic functionalityto manipulate lists.

The ListAdd method appends an item(SourceItemID, Attribute, Data array)to the list. Let's explain the algorithm. First, we find the top of thelist using the list name. Then, we seek the first blank cell underneaththe top cell. Once the destination is determine, the data is written tothe appropriate cells and the ICA is notified of the change. TheListCount method returns the number of items in the specified list. Thealgorithm works exactly like the ListAdd method but returns the totalnumber of items instead of inserting another element. The ListModifymethod replaces the specified item with the provided data. Let's explainthe algorithm. First, we find the top of the list using the list name.Second, we calculate the row offset based on the item number specified.Then, the ICA is notified of the removal of the existing item. Finally,the data related to the new item is written to the appropriate cells andthe ICA is notified of the change. The ListDelete method removes thespecified item. The algorithm works exactly like the ListModify methodbut no new data is added and the cells (width of the list set by ‘TotalColumns’) are deleted with the ‘move cells up’ parameter set to true.Keep this in mind, as designers often enter the wrong number of columnsin the Total Columns parameter. When they overestimate the TotalColumns, ListDelete will modify portions of the neighboring list, whichleads to erratic behavior when that list is displayed.

System Dynamics in Accordance with a Preferred Embodiment

To use system dynamics models in the architecture, an engine had to becreated that would translate student work into parameters for thesemodels. A complex system dynamics model to interact with an existingsimulation architecture is discussed below. The system dynamics modelprovides the following capabilities. Allow designers to build and testtheir system dynamics models and ICA feedback before the real interfaceis built. Reduce the programming complexity of the activities.Centralize the interactions with the system dynamics models. SystemDynamics Engine As with the simulation engine, the designer models thetask that he/she wants a student to accomplish using a Microsoft Excelspreadsheet. Here, however, the designer also creates a system dynamicsmodel (described later). The system dynamics engine will read all of thesignificant cells within the simulation model (Excel) and pass thesevalues to the system dynamics model and the ICA. After the systemdynamics model runs the information, the output values are read by theengine and then passed to the simulation model and the ICA.

FIG. 27 is a block diagram presenting the detailed architecture of asystem dynamics model in accordance with a preferred embodiment. Oncethe simulation model, system dynamics model and feedback are completelytested by designers, developers can incorporate the spreadsheet in agraphical user interface, e.g., Visual Basic as a development platform.FIG. 27 illustrates that when a student completes an activity, thevalues are passed to the system dynamics engine where the values arethen passed to the system dynamics model (as an input), written to thesimulation model and submitted to the ICA. When the system dynamicsmodel is played, the outputs are pulled by the engine and then passed tothe simulation model and the ICA. Note that the simulation model cananalyze the output from the system dynamics model and pass the resultsof this analysis to the ICA as well. The simulation model can then beread for the output values and used to update on-screen activitycontrols (such as graphs or reports). It is very important that allmodifications that the ICA and system dynamics model need to know aboutgo through the engine because only the engine knows how to call theseobjects. This significantly reduces the skill level required fromprogrammers, and greatly reduces the time required to program each task.In addition, the end-product is less prone to bugs, because the modeland tutor management will be centralized. If there is a problem, onlyone section of code needs to be checked. Finally, since the engine loadsthe data from the spreadsheet, the chance of data inconsistency betweenthe ICA, the system dynamics model and the application is insignificant.

The system dynamics model generates simulation results over time, basedon relationships between the parameters passed into it and othervariables in the system. A system dynamics object is used to integratewith Visual Basic and the spreadsheet object. The object includes logicthat controls the time periods as well as read and write parameters tothe system dynamics model. With Visual Basic, we can pass theseparameters to and from the model via the values in the simulationobject. The system dynamics object also controls the execution of thesystem dynamics model. What this means is that after all of theparameter inputs are passed to the system dynamics model, the engine canrun the model to get the parameter outputs. The system dynamics objectallows for the system dynamics models to execute one step at a time, allat once, or any fixed number of time periods. When the system dynamicsmodel runs, each step of the parameter input and parameter output datais written to a ‘backup’ sheet for two reasons. First, the range of datathat is received over time (the model playing multiple times) can beused to create trend graphs or used to calculate statistical values.Second, the system dynamics model can be restarted and this audit trailof data can be transmitted into the model up to a specific point intime. What this means is that the engine can be used to play asimulation back in time. When any event occurs within the systemdynamics engine, a log is created that tells the designers what valuesare passed to the simulation model, system dynamics model and ICA aswell as the current time and the event that occurred. The log is called“SysDyn.log” and is created in the same location as the applicationusing the engine. As with the spreadsheet object, the system dynamicsobject allows a large amount of the calculations to occur in the systemdynamics model and not in the activity code, again placing more controlwith the activity designers. Model objects are used to configure thesystem dynamics models with regard to the time periods played. Modelsare what the parameter inputs and parameter outputs (discussed later)relate to, so these must be created first. Every model has the followingapplication programming interface:

Data Field Name Type Description ModelID Long Primary Key for the tableTaskID Long TaskID of the task associated with the model ModelNameString*50 Name of the model (informational purposes) ModelDesc String*50Description of the model (informational purposes) SysDynModel String*50Filename of the actual system dynamics model Start Long Start time toplay model Stop Long Stop time to play model Step Long Interval at whichto play one model step and record data

This information is stored in the model table of the simulation database(ICASim.mdb). All of the values that will need to be manually entered bythe student that are passed into the system dynamics model areconfigured as parameter inputs (PInputs) objects. Every PInput has aninterface as detailed below.

Field Name Data Type Description PinputID long Primary Key for the tableTaskID long TaskID of the task associated with the parameter inputModelID long ID of the model associated with the parameter inputInputName string*50 Name of the parameter input (informational purposes)InputDesc string*255 Description (informational purposes) ReferenceNamestring*50 Name of the spreadsheet cell associated with the parameterinput¹ SimReferenceName string*50 Name of the associated parameter inthe system dynamics model TutorAware boolean Whether the ICA should benotified of any input CHANGES SourceItemID long SourceItemID of theparameter input TargetID long TargetID of the parameter input Row longSpreadsheet row number of the parameter input Column long Spreadsheetcolumn number of the parameter input SheetName string*50 Sheet name werethe parameter input is located

All of this information is stored for every parameter input in thePInput table of the simulation database (ICASim.mdb). PInputs consist ofone spreadsheet cell that can be populated by a designer at design timeor by the GBS application at run time via the system dynamics engineobject's methods. The purpose of the cell is to provide an entry pointto the simulation and system dynamics models. An example of an entrypoint would be the interest rate parameter in the interest calculationexample. The ICA is notified of any changes to the cell when anappropriate activity transpires. When the ICA is notified of a changetwo messages are sent to the ICA. The first is an ICANotifyDestroymessage with the parameter input information i.e., SourceItemID,TargetID and null as an attribute. This message is sent to inform theICA to remove information from its memory. The second message is anICANotifyCreate message with the parameter input information i.e.,SourceItemID, TargetID, Attribute (cell numeric value). This messageadvises the ICA to add this information to its memory. A PInput tablerecord in accordance with a preferred embodiment is presented below.

PInputID: 12345 TaskID: 123 ModelID: 1 InputName: Interest Rate inputInputDesc: Interest Rate input into interest calculation modelReferenceName: Interest_Rate SimReferenceName Param_Interest_RateTutorAware: True SourceItemID \1201 TargetID: 4001 Row: 6 Column: 3SheetName: Sheet1

Once the configuration is completed, the designer can also use the ICAUtilities to test the simulation. The Row, Column and SheetName valuesare automatically populated when the designer runs the parameters in theSystem Dynamics Workbench in the ICA Utilities. The followinginformation provides details describing the interaction components inaccordance with a preferred embodiment.

Title Description Procedural tasks (w/drag Tasks which require theconstruction of some kind of report with evidence drop) dragged anddropped to justify conclusions Procedural tasks (w/o drag New taskdesigns that are procedural in nature, have very little branching, anddrop) always have a correct answer. Ding Dong task Tasks that interruptthe student while working on something else. This template includesinterviewing to determine the problem, and a simple checkbox form todecide how to respond to the situation. Analyze and Decide (ANDIE) Mostcommonly used for static root cause analysis, or identification tasks.task Developed on SBPC as a result of 3 projects of experienceredesigning for the same skill. Evaluate Options (ADVISE) Used for tasksthat require learner to evaluate how different options meet stated goalsor requirements. Developed at SBPC after 4 projects experienceredesigning for the same skill. Does not allow drag drop as evidence.Run a company task Time based simulation where student “chooses ownadventure”. Each period the student selects from a pre-determined listof actions to take. Developed on SBPC as a simplified version of the BDMmanage task. Use a model task When user needs to interact with aquantitative model to perform what if analysis. May be used for dynamicroot cause analysis - running tests on a part to analyze stress points.ICA Dynamic Meeting Task Developed on BDM to mimic interaction stylesfrom Coach and ILS EPA. Supports dynamic-rule based branching - willscale to support interactions like EnCORE defense meetings and YES.Manage Task Time based simulation where student manages resources. HumanResources Management, managing a budget, manage an FX portfolio. QVIDStatic Meeting Task Developed on Sim2 to support agenda-driven meetingswhere user is presented with up to 5 levels of follow-up questions topursue a line of questioning. As they ask each question, it's follow-upsappear. Flow Chart Task Will support most VISIO diagrams. Developed onSim2 to support simple flow chart decision models. QVID Gather DataStatic flat list of questions to ask when interviewing someone. Not usedwhen Component interviewing skills are being taught (use QVID Staticmeeting task). Supports hierarchical questions and timed transcripts.Journalize Task Created to support simple journal entry tasks with up to2 accounts per debit or credit. New Complex Task A new task thatrequires a simulation componentThe system dynamics engine is the interface between the simulationmodel, the system dynamics model, the simulation database and theIntelligent Coaching Agent. The system dynamics engine is of interest tothe designer so that she can understand the mechanics of it. Once thedesigner has constructed the simulation model (Excel Spreadsheet), builtthe system dynamics model (PowerSim) and configured all of the parameterinputs and parameter outputs, a test can be performed using theworkbench included in the ICA Utilities (refer to ICA Utilitiesdocumentation). The developers, in turn, need to implement the calls tothe system dynamics engine in the GBS application that is being built.The following list identifies the files that need to be included in theVisual Basic project to use the system dynamics engine.

WSysDynEng.cls System dynamics Engine class wSysDynEng.bas Systemdynamics Engine module (this module was introduced only for speedpurposes because all the code should theoretically be encapsulated inthe class) wConst.bas Intelligent Coaching Agent constant declarationwDeclare.bas Intelligent Coaching Agent DLL interface wlca.clsIntelligent Coaching Agent class wlca.bas Intelligent Coaching Agentmodule (this module was introduced only for speed purposes because allof the code should theoretically be encapsulated in the class)

To utilize the system dynamics engine fully, the developer must placecode in different strategic areas or stages of the application. Initialstage—the loading of the form containing the simulation front-end. Thisis when the simulation model and system dynamic engine are initialized.Modification stage—Takes place when the user makes changes to thefront-end that impacts the simulation model PInputs). This is when theICA is notified of what's happening. Run stage—The system dynamics modelis run and parameter outputs are received. Feedback stage—The userrequests feedback on the work that they have performed. This is when thesimulation notifies the ICA of all output changes. Final stage—Thesimulation front-end unloads. This is when the simulation model issaved. These stages will be explained by including the Visual Basic codeinvolved as well as a short description of that code.

Initial Stage Code in Accordance with a Preferred Embodiment

1. Creating the ICA & the simulation engine objects: Code: SetmoSysDynEngine=New classSysDynEngine; Set moICA=New classICA;Description: The first step in using the system dynamics engine is tocreate an instance of the classSysDynEngine class and also an instanceof the class ICA class. Note that the engine and ICA should be modulelevel object “mo” variables. 2. Loading the simulation: Code:IRet=moSysDynEngine.OpenSimulation(FILE_SIM, Me.bookSim, True);IRet=moSysDynEngine.LoadSysDyn(mIICATaskID, DB_SIMULATION, 1);IRet=moSysDynEngine.LoadModel(MODEL_NAME, mbTaskStarted); Description:After the object creation, the OpenSimulation, LoadSimulation and LoadModel methods of the system dynamics engine object must be called. TheOpenSimulation method reads the specified Excel 5.0 spreadsheet file(FILE_SIM) into a spreadsheet control (bookSim). The LoadSysDyn methodopens the simulation database (DB_SIMULATION) and loads into memory alist of parameter inputs and a list of parameter outputs. The LoadModelmethod opens a system dynamics model (MODEL_NAME). Every method of thesystem dynamics engine will return 0 if it completes successfullyotherwise an appropriate error number is returned. 3. Initializing andloading the Intelligent Coaching Agent; Code:IRet=moICA.Initialize(App.Path & “\” & App.EXEName & “.ini”, App.Path &DIR_DATABASE, App.Path & DIR_ICADOC, App.Path & “\”);IRet=moICA.LoadTask(mIICATaskID, ICAStudentStartNew); Description: Thesystem dynamics engine only works in conjunction with the ICA. TheInitialize method of the ICA object reads the application .ini file andsets the Tutor32.dll appropriately. The LoadTask method tells the ICA(Tutor32.dll) to load the .tut document associated to a specific task inmemory. From that point on, the ICA can receive notifications. Note: Thetut document contains all the element and feedback structure of a task.Ex: SourcePages, SourceItems, TargetPages, Targets, etc. . . . 4.Restoring the simulation—Code:IRet=moSysDynEngine.RunPInputs(MODEL_NAME, True);IRet=moSysDynEngine.RunPOutputs(MODEL_NAME, True);IRet=moSysDynEngine.PassPInputsAll; Call moICA.Submit(0); CallmoICA.SetDirtyFlag(0, False) Description: Restoring the simulationinvolves many things: clearing all of the parameter inputs and outputswhen the user is starting over; loading the interface with data from thesimulation model; invoking the PassPInputsAll method of the systemdynamics engine object in order to bring the ICA to its original state;invoking the RunPInputs and RunPOutputs methods of the system dynamicsengine object in order to bring the system dynamics model to it'soriginal state; calling the Submit method of the ICA object to triggerthe ICA to play all of the rules; calling the SetDirtyFlag of the ICAobject to reset the user's session. Running parameters involves goingthrough the list of TutorAware PInputs and POutputs and notifying theICA of the SourceItemID, TargetID and Attribute value of every one.Modification Stage; 1. Reading parameter inputs & outputs; Code: DimsDataArray(2) as string; Dim vAttribute as variant; Dim ISourceItemID aslong, ITargetID as long; IRet=moSysDynEngine.ReadReference(“Input_Name”,vAttribute, ISourceItemID, ITargetID, sDataArray). Description: TheReadReference method of the system dynamics object will return theattribute value of the parameter input or output referenced by name andoptionally retrieve the SourceItemID, TargetID and related data. In thecurrent example, the attribute value, the SourceItemID, the TargetID and3 data cells will be retrieved for the parameter input named Input_Name.2. Modifying parameter inputs Code: Dim vAttribute as variant; DimISourceItemID as long; Dim sDataArray(2) as string; vAttribute=9999;sDataArray(0)=“Data Cell #1”; sDataArray(1)=“Data Cell #2”;sDataArray(2)=“Data Cell #3”;IRet=moSysDynEngine.WriteReference(“Input_Name”, vAttribute,sDataArray). Description: To modify a parameter input, call theWriteReference method of the system dynamics object and pass the PInputreference name, the new attribute value and optionally a data array (anadditional information to store in the simulation model). The systemdynamics engine notifies the ICA of the change. Run Stage 1. Playing theSystem Dynamics Model; Code:IRet=moSysDynEngine.PlayModel(SYSDYN_PLAYSTEP);IblCurrentTime.Caption=moSysDynEngine.CurrentTime; andIblLastTime.Caption=moSysDynEngine.LastTime; Description: Playing thesystem dynamics model is also handled by the system dynamics engine.There are three ways that the models can be played, all at once, onestep at a time (shown above) or until a specific point in time. Theseare the parameters that are passed into the PlayModel method. Playing ofthe model generates the parameter output values and passes the TutorAware POutputs to the ICAT. The engine also keeps track of time andthese values can be read using the CurrentTime and LastTime properties.2. Jumping Back in a System Dynamics Model Code:IRet=moICA.LoadTask(mIICATaskID, ICAStudentStartNew);IRet=moSysDynEngine.JumpBack(TIME_TO_JUMP_TO). Description: Because thesystem dynamics engine writes backup copies of the parameters passed toand from it, it can start over and resubmit these values back to thesystem dynamics model until a given period of time. To do this, the codewould need to restart the ICA and then call the system dynamics engineto jump back to a given time (TIME_TO_JUMP_TO). Feedback stage 1.Triggering the ICA Rule engine; Code: IRet=moICA.Submit(ICoachID);Description: Once the simulation has been processed, the Submit methodof the ICA object must be called to trigger all the rules and deliverthe feedback. This feedback will be written by the Tutor32.dll to twoRTF formatted files. One file for previous feedback and one file for thecurrent feedback.

ICA Configuration in Accordance with a Preferred Embodiment

FIG. 28 is an overview diagram of the logic utilized for initialconfiguration in accordance with a preferred embodiment. Since thestructure of the feedback is the same as other on-line activities, theICA can also be configured in the same manner. For ease of creation andmaintenance of ICA feedback, it is recommended that the feedback isconstructed so that only one rule fires at any point in time. Note thatthe organization of the example is one of many ways to structure thefeedback. Step 1: Create a map of questions and follow-up questions;Before designers start configuring the ICA, they should draw a map ofthe questions, videos and follow-up questions that they wish to use inthe on-line meeting. This will give them a good understanding of theinteractions as they configure the ICA. Step 2: Create a coach; Allfeedback is given by a coach. Create a specific coach for the on-linemeeting. Step 3: Create the Source Items and Targets

Every question will have one Source Item (1) and Target (2) associatedwith it. These will be used by the ICA to show videos and follow-upquestions. For organizational purposes and ease of reading, it isrecommended that each Source Page (“0 Intro”) contain all of the followup questions (“Intro Q1”, “Intro Q2”, “Intro Q3”). Targets can becreated one per Source Item (shown here) or one per many Source Items.This is not very important, so long as there are distinct Source Itemand Target associations. Once the Source Items and Targets have beencreated, associate them into SourceItemTargets (3) and give them arelevance of one. These are the unique identifiers which the ICA willuse to fire rules and to provide feedback to the student. Step 4: Createthe Parent Header (Video Information) FIG. 29 is a display of videoinformation in accordance with a preferred embodiment. Feedback (CoachItems) are organized into Target Groups (1). In FIG. 29, each on-linequestion has one Target Group for ease of maintenance. Each TargetGroupmust have at least one related Target (4). These are theSourceItemTarget mappings that were made at the end of Step 3. Next,Rules (2) are created to fire when the SourceItemTarget is mapped (aquestion is clicked). Coach Items (3) are associated to a rule andrepresent the feedback which will be shown if the rule is fired. The ICAUtilities incorporate business simulation into a multimedia application.What this means is that there is now a middle layer between theapplication and the ICAT. These utilities, along with the simulationengine (described later), allow the architecture to be a front end tothe simulation. Now, any changes to a simulation model do not need to beincorporated into code. The ICA Utilities and simulation engine workwith simulation models created in Microsoft Excel. After the model iscreated, the designer uses the Defined Name function in Excel to flagspecific cells that are to be used by the application and the ICAUtilities in accordance with a preferred embodiment. FIG. 30 illustratesan ICA utility in accordance with a preferred embodiment. The ICAUtilities consist of six utilities that work with the IntelligentCoaching Agent Tool (ICAT) to incorporate business simulation with themultimedia application.

1. A computer-readable medium having computer-executable instructionsfor creating a presentation to a user, comprising: (a) presentinginformation indicative of a goal, the goal being associated with a skillrequired for the user in a business endeavor; (b) integratinginformation that motivates accomplishment of the goal; (c) monitoringprogress toward the goal and providing feedback that further motivatesaccomplishment of the goal, wherein: the feedback is characterized by aset of profiles and topics; the profiles trigger the topics in a concepttree to obtain a plurality of activated topics; and the feedback isselected from the plurality of activated topics in the concept tree by:identifying a top-most target group with an activated topic; if thetop-most target group is a first type of feedback, selecting thatfeedback for display to the user without examining any other activatedtopics; if the top-most target group is a second type of feedbackdifferent from the first type of feedback, grouping the activatedfeedback in the children target groups for display to the user; (d)receiving an activity description and a corresponding activity log, theactivity description being descriptive of an improper operation of thecomputer-executed instructions associated with at least one usersubmission incurred during the presentation, the improper operationbeing detected by the user from the feedback, the corresponding activitylog being indicative of the at least one user submission correlated tothe improper operation and the feedback associated with the at least oneuser submission; (e) performing regression analysis of thecomputer-executable instructions as the presentation subsequentlyexecutes, wherein the regression analysis utilizes the activitydescription and the corresponding activity log to recreate the improperoperation; and (f) in response to (e), altering the set of profiles andtopics based on a developer input to correct the feedback by adding anadditional profile that triggers an additional topic to the user whenthe additional profile is activated, the additional profile including aset of conditions met by work submitted by the user, the altered set ofprofiles and topics addressing a specific combination of mistakesperformed by the user as monitored in (c).
 2. The computer-readablemedium as recited in claim 1, including instantiating a particularfeedback model based on the regression analysis.
 3. Thecomputer-readable medium as recited in claim 1, including receiving andanalyzing user responses using an expert system to determine details ofthe computer-executable instructions to store.
 4. The computer-readablemedium as recited in claim 1, including storing user interactions withthe presentation as the presentation executes.
 5. The computer-readablemedium as recited in claim 1, including playing back user interactionswith the presentation.
 6. The computer-readable medium as recited inclaim 1, including printing a report based on the regression analysis.7. The computer-readable medium as recited in claim 1, includingorganizing the regression analysis information in a company preferredmanner to facilitate performance metrics for the company.
 8. Thecomputer-readable medium as recited in claim 1, including presenting atailored simulation based on the regression analysis as the presentationexecutes.
 9. The computer-readable medium as recited in claim 1,including presenting a tailored simulation based on the regressionanalysis as the presentation executes.
 10. An apparatus that creates apresentation to a user, comprising: (a) a processor that executes acomputer program to support the presentation; (b) a memory that storesinformation under the control of the processor; (c) logic that presentsinformation indicative of a goal, the goal being associated with a skillrequired for the user in a business endeavor; (d) logic that integratesinformation that motivates accomplishment of the goal; (e) logic thatmonitors progress toward the goal and provides feedback that furthermotivates accomplishment of the goal, wherein: the feedback ischaracterized by a set of profiles and topics; the profiles trigger thetopics in a concept tree to obtain a plurality of activated topics; andthe feedback is selected from the plurality of activated topics in theconcept tree by: identifying a top-most target group with an activatedtopic; if the top-most target group is a first type of feedback,selecting that feedback for display to the user without examining anyother activated topics; if the top-most target group is a second type offeedback different from the first type of feedback, grouping theactivated feedback in the children target groups for display to theuser; (f) logic that receives an activity description and acorresponding activity log, the activity description being descriptiveof an improper operation of the apparatus associated with at least oneuser submission incurred during the presentation, the improper operationbeing detected by the user from the feedback, the corresponding activitylog being indicative of the at least one user submission correlated tothe improper operation and the feedback associated with the at least oneuser submission; and (g) logic that performs regression analysis of thecomputer program as the presentation subsequently executes and thatalters the set of profiles and topics to correct the feedback providedby the logic in (e) by adding an additional profile that triggers anadditional topic to the user when the additional profile is activated,the additional profile including a set of conditions met by worksubmitted by the user, wherein the regression analysis utilizes theactivity description and the corresponding activity log to recreate theimproper operation and wherein the altered set of profiles and topicsthat are based on a developer input addresses a specific combination ofmistakes performed by the user as monitored in (e).
 11. An apparatusthat creates a presentation as recited in claim 10, including logic thatinstantiates a particular feedback model based on the regressionanalysis.
 12. An apparatus that creates a presentation as recited inclaim 10, including logic that receives and analyzes user responsesusing an expert system to determine details of the computer program tostore.
 13. An apparatus that creates a presentation as recited in claim10, including logic that store user interactions with the presentationas the presentation executes.
 14. An apparatus that creates apresentation as recited in claim 10, including logic that plays backuser interactions with the presentation.
 15. An apparatus that creates apresentation as recited in claim 10, including logic that prints areport based on the regression analysis.
 16. An apparatus that creates apresentation as recited in claim 10, including logic that organizes theregression analysis information in a company preferred manner tofacilitate performance metrics for the company.
 17. An apparatus thatcreates a presentation as recited in claim 10, including logic thattailors feedback based on regression analysis as the presentationexecutes.
 18. An apparatus that creates a presentation as recited inclaim 10, including logic that presents a tailored simulation based onthe regression analysis as the presentation executes.
 19. Acomputer-readable medium having computer-executable instructions forperforming steps comprising: (a) presenting information indicative of agoal, the goal being associated with a skill required for a user in abusiness endeavor; (b) integrating information that motivatesaccomplishment of the goal; (c) monitoring progress toward the goal andproviding feedback that further motivates accomplishment of the goal,wherein: the feedback is characterized by a set of profiles and topics;the profiles trigger the topics in a concept tree to obtain a pluralityof activated topics; and the feedback is selected from the plurality ofactivated topics in the concept tree by: identifying a top-most targetgroup with an activated topic; if the top-most target group is a firsttype of feedback, selecting that feedback for display to the userwithout examining any other activated topics; if the top-most targetgroup is a second type of feedback different from the first type offeedback, grouping the activated feedback in the children target groupsfor display to the user; (d) receiving an activity description and acorresponding activity log, the activity description being descriptiveof an improper operation of the computer-executable instructionsassociated with at least one user submission incurred during apresentation to the user, the improper operation being detected by theuser from the feedback, the corresponding activity log being indicativeof the at least one user submission correlated to the improper operationand the feedback associated with the at least one user submission; (e)performing regression analysis of the computer-executable instructionsas the presentation subsequently executes, wherein the regressionanalysis utilizes the activity description and the correspondingactivity log to recreate the improper operation; and (f) in response to(e), altering the set of profiles and topics based on a developer inputto correct the feedback by adding an additional profile that triggers anadditional topic to the user when the additional profile is activated,the additional profile including a set of conditions met by worksubmitted by the user, the altered set of profiles and topics addressinga specific combination of mistakes performed by the user as monitored in(c).
 20. The computer-readable medium of claim 19 havingcomputer-executable instructions to perform: (g) initiating a processingof the notification from the user by an indicator being inserted in afeedback display.